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Executive Summary 



This report presents the results of a June 1997 onsite evaluation of a U.S. Department of 
Education HEA II-B grant awarded to the Portland Area Library System (PORTALS) during the period 
October 1, 1995 to September 30 1997. In addition to PORTALS, grant participants include: Multnomah 
County Library (MCL), Oregon Historical Society (OHS), and the Oregon State Library (OSL). The 
Onsite Evaluator was Joe Ryan. Dr. Charles R. McClure, the Evaluation Consultant, reviewed Ryan’s 
report and the report was reviewed by, and discussed with PORTALS interim Executive Director, Dr. 
Jim Kopp. The goal of this onsite evaluation was to summarize for PORTALS and mirror for grant 
participants their activities in key areas (identified above) so as to contribute to their own ongoing 
self-assessment. The report looks at each grant participant s activities covering: 

• Project Objectives: This section identifies the initial project objectives (using text from the 
grant proposal) and summarizes activity to date. 

• Chronology of Events and Accomplishments to Date: This section highlights, in 
chronological fashion, key project activities, milestones, outcomes, and accomplishments to 
date given the project s objectives. Omitted are regular, ongoing, phone conversations or 
meetings at various conferences between the Evaluation Consultant and PORTALS staff or 
project participants. 

• Staff Development Activities: This section summarizes efforts by grant participants to 
obtain education and training for staff to better implement the grant objectives. 

• Principal Project Work Products: This section lists and describes the major products, 
services, and other outcomes as a result of the grant. 

• Dissemination of Project Results: This section identifies specific ways grant participants 
disseminated project findings to the local, state, and national communities and the 
profession including publications, videos, press releases, presentations, external training 
sessions, etc. 

• Evaluation Activities: This section identifies the specific ways each participant 
evaluated the project. 

• Lessons Learned: This section identifies significant lessons learned by project participants 
which may also be of interest to similar organizations in other settings. 

• Next Steps: This section suggests next steps to be taken as this grant period ends and to 
advance the project s objectives after the funding period. 

• Recommendations: This section contains the evaluators recommendations for improvement 
based on the onsite visit. 

The report closes with the evaluators overall conclusions and recommendations to PORTALS. The grant 
specifically does not examine PORTALS non-grant activities, PORTALS members (other than those 
who directly participated in the grant), and PORTALS internal management, nor is the evaluation a 
financial audit or examination of accounting practices related to the grant. 
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Method 



The consultant divided his activities into three principal phases: 

• Preparation for data collection: including reading of available documentation; 
determination of evaluation objectives and construction of method and interview protocols 
in consultation with the Evaluation Consultant and PORTALS Executive Director prior to 
his departure; an evaluation of the project participants web sites (see Appendix P-1); 
logistical arrangements; preparation of interview protocols; and, drafting of an initial 
report indicating what was tentatively known. 

• Onsite Visit June 3 - June 20 1997: The Onsite Evaluator visited each grant participant in 
order to collect data including initial interviews and focus groups at each site and 
subsequent follow-up as appropriate. The Onsite Evaluator conducted approximately 
thirty interviews during his visit. For a complete schedule of interviews see Appendix P-2. 
A key informant at each site checked a draft of the basic factual findings for accuracy. 

• Data Analysis and Final Report Preparation: The Onsite Evaluator and Evaluation 
Consultant examined the results of data collection activities and prepared a draft report 
submitted July 7, 1997 to the interim Executive Director. 

The interim Executive Director, Evaluation Consultant, and Onsite Evaluator conducted a conference 
call on July 22, 1997 to review the draft report. The evaluators submitted this final report on August 7 
1997. 



The reader is referred to the detailed findings and recommendations provided for each of the 
grant participants in the body of the report. In general, this report documents substantially improved 
citizen access to federal, state, county, and city government information in Oregon as the result of the 
U.S. Department of Education HEA II-B grant to the Portland Area Library System (PORTALS). This 
grant enabled the: 

• Conversion of important historic government and other materials to digital format, 

• Creation of new government information in digital format, 

• Organization of government information in more useful ways for citizen access, 

• Change in internal institutional practice necessary to better deliver government 
information to citizens via electronic networks, 

• Increased capacity to deliver information via world wide web pages on electronic networks, 

• Ability to receive electronic government information extended to rural Oregon public 
libraries. 

The improved access to government information is due to the significant progress made to date by 
Oregon State Library, Multnomah County Library, and the Oregon Historical Society toward meeting 
their grant objectives. PORTALS itself will need to make additional efforts during the remaining grant 
period in order to achieve its own programmatic objectives under the grant. 



I. Introduction 



The Portland Area Library System (PORTALS) administers and participates in a U.S. 
Department of Education HEA II-B grant, "Citizen access to government and other information" during 
the period October 1, 1995 to September 30 1997. 1 In addition to PORTALS, grant recipients include 
Multnomah County Library (MCL), Oregon Historical Society (OHS), and the Oregon State Library 
(OSL). 2 This report presents one element of the evaluation component of the grant, the results of an 
onsite evaluation of the project conducted by Joe Ryan during June 1997, reviewed by Dr. Charles R. 
McClure, project Evaluation Consultant, and discussed with interim Executive Director, Dr. Jim Kopp. 
The purpose of the external evaluation was to examine at each site: 

• Project Objectives: Identify initial project objectives and summarizes activity to date. 

• Chronology of Events and Accomplishments to Date: Highlight, in chronological fashion, 
key project activities, milestones, outcomes, and accomplishments including principal work 
products to date given the projects objectives. 

• Staff Development Activities: Note efforts by grant participants to obtain education and 
training for staff to better implement the grant objectives. 

• Principal Project Work Products: List and describe the major products, services, and other 
outcomes as a result of the grant. 

• Dissemination of Project Results: Identify specific ways grant participants disseminated 
project findings to the local, state, and national communities and the profession including 
publications, videos, press releases, presentations, external training sessions, etc. 

• Evaluation Activities: Identify specific ways each participant used to evaluate the project 
both internally and using user evaluations. What conclusions did each site make based on 
these evaluation techniques? What does each project participant think about the utility of 
their evaluation methods to date? Identify specific plans for future evaluation? 

• Lessons Learned: Identify lessons learned by project participants of potential interest to 
similar organizations in other settings. 

• Next Steps: Suggest next steps to be taken to advance the project's objectives after the 
funding period. 

The onsite evaluation was specifically not an examination of PORTALS members (other than the grant 
activities of those who directly participated in the grant), PORTALS non-grant activities or internal 
management, nor was the evaluation a financial audit of grant funds expended by PORTALS or grant 
recipients, nor was there any evaluation of the subsequent HEA II-B grant installments to PORTALS. 

The goal of this onsite evaluation was to summarize for PORTALS and mirror for grant 
participants their activities in key areas (identified above) so as to contribute to their own ongoing 
self-assessment. The consultant divided his activities into three principal phases: 



1 Subsequently referred to as "the grant" throughout the report. 

2 Subsequently referred to as the "grant participants" throughout the report. 
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• Preparation for data collection: including reading of available documentation; 
determination of evaluation objectives and construction of method and interview protocols 
in consultation with the Evaluation Consultant and PORTALS Executive Director prior to 
his departure; an evaluation of the project participants web sites; logistical arrangements; 
preparation of interview protocols; and, drafting of an initial report indicating what was 
tentatively known. 

• Onsite Visit June 3 - June 26 1997: The Onsite Evaluator visited each grant participant in 
order to collect data including initial interviews at each site and subsequent follow-up as 
appropriate. The Onsite Evaluator conducted approximately thirty interviews during his 
visit. A key informant at each site checked a draft of the basic factual findings for 
accuracy. 

• Data Analysis and Final Report Preparation: The Onsite Evaluator and Evaluation 
Consultant examined the results of data collection activities and prepared a draft report 
submitted July 7, 1997 to the interim Executive Director. 

The interim Executive Director, Evaluation Consultant, and Onsite Evaluator conducted a telephone 
conference call on July 22, 1997 to review the draft report. The evaluators submitted this final report on 
August 12, 1997. 

This report has the following organization. Background information on the grant and the 
evaluation component specifically, appear next. Then a brief section describing the method used to 
conduct the evaluation follows. The evaluation findings appear next, arranged by major project 
participant, i.e., PORTALS, OSL, MCL, and OHS. Each participant's section includes: project 
objectives and outcomes in brief, evaluator's activities, project chronology of events and 
accomplishments, staff development activities, project work products, dissemination efforts, 
evaluation efforts, lessons learned, next steps, and the evaluators' recommendations for each specific 
grant participant. The report concludes with overall recommendations by the evaluation team based on 
the findings from the onsite evaluation. The authors wish to acknowledge and thank the many project 
participants who provided information and otherwise made this evaluation possible (for a list of 
contacts see Appendix P-3). Special thanks to the key liaison personnel at each site used by the 
evaluator to coordinate site visits and cross-check findings including Rushton Brandis (OSL - 
JumpStart), Ernest Perez (OSL), Donna Reed (MCL), Sue Seyl (OHS), Karen Starr (PORTALS), and 
Todd Welch (OHS). 

Background 

In 1993, a group of the principal libraries in the Portland, Oregon region established the 
Portland Area Library System (PORTALS). The PORTALS web page <http://www.portals.org/> 
concisely state the organization's purpose: "PORTALS is an organization of sixteen public and private 
institutions committed to working cooperatively in order to expand and enrich the information resources 
and services necessary for the scholarly research activities of people in the greater Portland metro 
area." PORTALS provides a website (funded by the grant) with links to member institutions, their 
libraries, and their automated library catalogs; access to selected licensed electronic databases; 
information about reciprocal borrowing among members, and a variety of other services for its members. 
PORTALS members include: Clark College, George Fox University, Lewis & Clark College, Linfield 
College, Marylhurst College, Mount Hood Community College, Multnomah County Library (grant 
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participant), Oregon Graduate Institute of Science and Technology, Oregon Health Sciences University, 
Oregon Historical Society (grant participant). Pacific University, Portland Community College, 
Portland State University, Reed College, University of Portland, and Washington State University - 
Vancouver. The State of Oregon and member institutions provide funding for PORTALS. 

On February 6, 1995 PORTALS applied for a U.S. Department of Education (DoEd) grant. DoED 
subsequently awarded a grant for the period October 1, 1995 - September 30, 1997. The Department of 
Education also awarded subsequent HEA II-B grants to PORTALS all of which DoED treats as one 
contract with PORTALS. However, the present evaluation is only for the first award (for the period 
October 1, 1995 - September 30, 1997). The first grant evaluated here had several objectives including: 

• Deployment by PORTALS of a state-of-the-art, high bandwidth, multi-media capable 
network linking PORTALS members. 

• Provision of a range of information resources and services accessible by PORTALS members 
with initial emphasis on local, state, and federal government information. These resources 
and services were either to be created locally, purchased and made accessible to PORTALS 
members, or identified and linked to the PORTALS web server. 

• Development of ongoing mechanisms for formative and summative evaluation of network, 
information resources, and services. Formative evaluation would enable the grant 
participants to adjust their network, staff, resources, and services to meet its users’ needs. 
Summative evaluation would document, in snapshot fashion, the grant participants’ 
progress toward their objectives. 

The grant offered an opportunity to increase collaboration among PORTALS members via the activities 
of network deployment, information resources and services provision, and collective formative and 
summative evaluation. 

In early 1995, PORTALS Director of Network Development, Millard Johnson hired Dr. Charles 
R. McClure as a consultant to help PORTALS prepare the programmatic evaluation component of the 
grant proposal. Subsequently, Howard McGinn became the PORTALS Executive Director. Dr. McClure 
assumed the role of the grant project Evaluation Consultant with the funding of the grant in October 
1995. In February 1996 after extensive consultation with project participants Dr. McClure developed a 
more detailed evaluation plan (See Appendix P-4). The evaluation plan stressed self assessment by 
participating PORTALS members, employment of a grant-funded, locally-based research assistant to 
conduct onsite data collection, with McClure advising as necessary. For various reasons, a locally- 
based research assistant could not be found or hired. This prompted a further revision to the evaluation 
plan after a January 22-23, 1997 site visit by Dr. McClure and discussion with the then PORTALS 
Executive Director, Howard McGinn (see Appendix P-5). The U.S. Department of Education 
subsequently approved this revision ( see Appendix P-6). The Evaluation Consultant asked grant 
participants to submit self-evaluation reports in February 1997. Each of the grant participants filed an 
evaluation report with the exception of PORTALS itself (who did not file a report for reasons not 
known to the evaluators). For most of the participants, this was the first formal evaluation of their 
grant activities, conducted 15 months into the two year funding cycle. As part of the evaluation 
revision, PORTALS, with DOEd approval, hired Joe Ryan, a co-author of this document, to conduct an 
onsite evaluation which is the subject of the present report. 
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On March 31 , 1997 Howard McGinn resigned his position as PORTALS Executive Director. On 
May 16, 1997 Dennis Gilbert, the Project Systems Programmer, resigned. Jim Kopp became interim 
Executive Director on June 2, 1997. Joe Ryan conducted his onsite visit and interviews June 3-17, 19-20, 
and 25, 1997. The evaluators filed a preliminary draft of the onsite evaluation report with the interim 
Executive Director on July 7, 1997. The interim Executive Director reviewed the report and discussed it 
with the Evaluation Consultant and Onsite Evaluator in a telephone conference call on July 22, 1997. 

The Evaluators filed a final report with the PORTALS interim Executive Director on August 12, 1997. 

At present, the PORTALS staff consists of Jim Kopp, the interim Executive Director and Karen 
Starr, Director of Network Information. Tom Pfingsten became HEA II-B grant principal investigator 
(a position previously filled simultaneously by the Executive Director) in June 1997 and hired Lois 
Cohen as Grants Assistant to report to the principal investigator. The PORTALS Board agreed to 
replace the Project Systems Programmer with a Systems/Network Administrator (for job announcement 
see WWW: http://www.portals.org/sysadmin.html). The management of various licensed databases 
made available to PORTALS members (a non-grant activity) took an inordinate amount of the former 
Project Systems Programmer time hampering completion of certain components of the grant. A plan to 
move to vendor supported, Internet accessible licensed databases is nearing approval. PORTALS will 
create a technology advisory group to be appointed from members of PORTALS library automation and 
computer center staffs to assist the Project Systems Programmer and interim Executive Director in 
various grant activities. 

A draft strategic planning document (Available: WWW: 
http:/ /www.portals.org/plantoplan.html) announced after the onsite evaluator's visit notes that: 
"Over the next 12 months, PORTALS will engage in a strategic planning process. The goal of that 
process is to determine the development of the PORTALS consortia library and information 
environment in the metro area through the year 2000." The present report is opportune as PORTALS re- 
thinks its role and focuses on the scholarly information needs of its members. The present report of the 
onsite evaluation visit seeks to contribute to PORTALS strategic planning effort by documenting the 
activities associated with one of PORTALS major activities in the recent past and by offering 
suggestions based on the grant experience. 

Onsite Evaluator's Biography in Brief 

Joe Ryan is presently completing his Ph.D. under Dr. Charles R. McClure at Syracuse 
University's School of Information Studies. He has experience as a consultant and evaluator of projects 
in libraries, information centers, government organizations, non-profits, and corporations. Recent clients 
include EMF/World Bank; UN Food and Agriculture Organization; various Federal agencies such as the 
National Archives, Office of Management and Budget, Office of Technology Assessment; and, the State 
Library of North Carolina. He is also an author and publisher of a set of guides to International, U.S. 
Federal, State and Local government information on the Internet. Ryan received an M.L.S. from 
Syracuse University's School of Information Studies in 1978 and has built, worked in, or managed 
public, special, and academic libraries since then. 
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Method 



Joe Ryan conducted the onsite evaluation after consultation with Dr. McClure, and the previous 
Executive Director, Howard McGinn. As a courtesy, the Evaluators kept Jim Kopp apprised of the 
Onsite Evaluator's plans prior to the interim Executive Director's formal appointment. The goal of this 
onsite evaluation was to summarize for PORTALS and mirror for grant participants their activities in 
key areas (identified above) so as to contribute to their own ongoing self-assessment. The consultant 
divided his activities into three principal phases: 

• Preparation for data collection: including reading of available documentation; 
determination of evaluation objectives and construction of method and interview protocols 
in consultation with the Evaluation Consultant and PORTALS Executive Director prior to 
his departure; an evaluation of the project participants web sites (see Appendix P-1); 
logistical arrangements; preparation of interview protocols; and, drafting of an initial 
report indicating what was tentatively known. 

• Onsite Visit June 3 - June 26, 1997: The Onsite Evaluator visited each grant participant in 
order to collect data including initial interviews at each site and subsequent follow-up as 
appropriate. The Onsite Evaluator conducted approximately thirty interviews during his 
visit. A key informant at each site checked a draft of the basic factual findings for 
accuracy. 

• Data Analysis and Final Report Preparation: The Onsite Evaluator and Evaluation 
Consultant examined the results of data collection activities and prepared a draft report 
submitted July 7, 1997 to the interim Executive Director. 

The interim Executive Director, Evaluation Consultant, and Onsite Evaluator conducted a telephone 
conference call on July 22, 1997 to review the draft report. The evaluators submitted this final report on 
August 12, 1997. 

Efforts to Ensure Data Quality 

Field evaluation is an art requiring quick assessment of opportunities and dangers to data 
quality on site. As Schatzman & Strauss (1973, p. vii) note: 

...much of the research process consists of dealing with a flow of substantive discoveries and 
with field contingencies that variously modify the research; therefore the researcher is 
constantly attentive to options which are circumstantially presented to him, or which are 
created by him. Thus the field researcher is depicted as a strategist ; for without linear- 
specific design - the researcher must develop procedure as he goes. 

But field research is also a science, involving the systematic effort to reduce error due to researcher 
bias, incomplete or inaccurate data, and a host of other causes. 

The Evaluators took a number of steps to reduce the threats to data quality in the present 
evaluation, both during data collection and later during analysis (as suggested by Guba & Lincoln, 1981; 
Miles & Huberman, 1994; Patton, 1990; and Schatzman & Strauss, 1973) including: 
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• Pre-structured research questions and interview instruments and pre-planned fieldwork. 

• Chose standard, well-regarded, methods familiar to the evaluator and appropriate to the 
setting (McClure, 1994). Primary methods were qualitative (Miles & Huberman, 1994) 
including the use of documentary evidence, interviews (Spadley, 1979), focus groups 
(Kruger, 1988 and Morgan, 1988) and preparation of case studies (Stake, 1994 and Yin, 1994) 

• Documented fully, research design decisions in writing and in discussions with the 
Evaluation Consultant. 

• Sought dis-confirming and outsider evidence and points of view actively. Attempted, 
within the constraints of the visit, to interview stakeholders from multiple-perspectives. 

• Responded flexibly to the new and unexpected opportunities the data offer. 

• Documented fully the data collected. Where possible, the Onsite evaluator tape recorded 
interviews while maintaining interviewee confidentiality. Evaluators conducted follow- 
up interviews where necessary. 

• Triangulated the data collected and used mixed methods. Data collected from 1 one source 
was cross-checked with another. The Evaluators compared data collected using one method 
with answers obtained via another method. The Evaluators shared drafts of factual 
portions of the final report with a key liaison at each site to check for accuracy. 

• Pre-structured data analysis and reporting as suggested by Miles & Huberman (1994). This 
approach was possible because most of the data collection was pre-structured and the 
intended shape of the final report was known. 

• Checked the quality of the evidence by tracking the chain of evidence gathered to be sure it 
was firm enough to support statements made. 

Each of these efforts and others increased the validity and reliability of the evaluation findings and 
provide a firm basis for making recommendations. 

Organization of the Report 

An evaluation of each grant participants project will be presented in subsequent sections of the 
report. Grant participants are PORTALS itself; the Oregon State Library, its access to state government 
information effort and Project JumpStart grants to connect rural libraries to the Internet; the Multnomah 
County Library's efforts with local governments and the Internet; and, the Oregon Historical Society’s 
efforts to digitize and make more accessible portions of its significant collection of Oregon materials via 
the Internet. 

The report discusses the following areas for each grant participant evaluated: program 
objectives using language from the original grant proposal and outcomes to date in brief, the evaluator's 
activities in order to compile each portion of the report, a chronological summary of key events and 
accomplishments, staff development activities, project work products, efforts to disseminate project 
results, evaluation efforts, lessons learned, next steps, and the evaluators’ recommendations. 
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This report documents substantially improved citizen access to federal, state, county, and city 
government information in Oregon as the result of the U.S. Department of Education HEA II-B grant to 
the Portland Area Library System (PORTALS). This grant enabled the: 

• Conversion of important historic government and other materials to digital format, 

• Creation of new government information in digital format, 

• Organization of government information in more useful ways for citizen access, 

• Change in internal institutional practice necessary to better deliver government 
information to citizens via electronic networks, 

• Capacity improvements to deliver information via world wide web pages on electronic 
networks, and, 

• Extension of the ability to receive electronic government information to rural Oregon public 
libraries. 

This is due to the significant progress made to date by Oregon State Library, Multnomah County 
Library, and the Oregon Historical Society toward meeting their grant objectives. PORTALS itself 
must make additional efforts during the remaining grant period in order to achieve its own 
programmatic objectives under the grant. 
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II. Portland Area Library System (PORTALS) 



This section of the report examines the PORTALS programmatic component of the HEA II-B 
grant including: program objectives using language from the original grant proposal and outcomes to 
date in brief, the evaluator's activities in order to compile this section of the report, a chronological 
summary of key events and accomplishments, staff development activities, project work products, 
efforts to disseminate project results, PORTALS evaluation efforts, lessons learned, next steps, and the 
evaluators' recommendations. 

Project Objectives and Outcomes in Brief 

This sub-section identifies initial project objectives (using text from the grant proposal) for the 
PORTALS portion of the HEA II-B grant and summarizes progress to date. 

PORTALS will create a backbone network and provide supported access to government 
information resources over this network to its members. Specifically, PORTALS will: 

• Create a PORTALS backbone network: This network will consist of a central PORTAL’S 
server networked via T1 lines provided by Northwest Net to fourteen servers located at 
each participating PORTALS institution with fourteen public multimedia workstations 
attached to these servers (1 at each institution). This network will in turn be accessible to 
other participants with some restrictions relating to copyright and vendor licensing. 

At present, a PORTALS server and web page (http://www.portals.org/) is available via 
the Internet. But the PORTALS backbone network, a key component of the PORTALs 
portion of the grant, is not deployed. Further, PORTALS deployment during the grant 
period reviewed here is in question. The former Project Systems Programmer purchased and 
tested the hardware and developed software necessary to deploy the network prior to his 
resignation. The equipment needs to be installed at the PORTALS members sites, staff at 
these sites trained, and the maintenance programs tested. In addition, there is uncertainty 
as to the commitment of staff time necessary to maintain the backbone network once 
deployed. The present PORTALS interim Executive Director indicates that the 
deployment of the backbone network is a priority. PORTALS posted advertisements for a 
replacement for the Project Systems Programmer. Technically supporting the databases 
offered to the PORTALS membership took a significant portion of the former Project 
Systems Programmer time. A plan is in place to remove PORTALS from the database 
support business. Instead, PORTALS members will access vendor supported databases via a 
PORTALS authentication gateway. PORTALS hopes this will free the new Project Systems 
Programmer time for other activities including deployment of the backbone network. A 
technology advisory group to be appointed from members of PORTALS library automation 
and computer center staffs will be named to assist Project Systems Programmer and interim 
Executive Director in various grant activities. 

• Provide user friendly and supported access to government information and other resources 
over the backbone network: The purpose of the backbone network is to provide the 
PORTALS membership with access to such content as: government sources of information 
(produced in part by PORTALS members the Oregon Historical Society and Multnomah 
County Library and funded in part by the grant), the PORTALS licensed databases 
(AIDSLINE, BIOSIS Previews, Business Abstracts, CINAHL, Compendex, Dissertation 
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Abstracts, ERIC, HealthSTAR, and MEDLINE, MLA Bibliography, Newspaper Abstracts, 
Periodical Abstracts Research II, PsychINFO, UnCover, and WORLDCAT), PORTALS 
administrative information, and other information of interest to the PORTALS 
membership. The provision of this content should be via a user-friendly interface and be 
supported by user training, staff development, and a virtual helpline. 

Although a PORTALS backbone network is not deployed, PORTALS members can access a 
PORTALS website (http://www.portals.org/). The web page provides PORTALS members 
with access to PORTALS licensed databases, adequate basic links to federal, state, and 
county and city government information (including government information generated by 
PORTALS members funded by the grant), and PORTALS administrative information. 

The web page user interface is adequate. However PORTALS did not conduct any user 
training or staff development related to the grant for the PORTALS membership, and did 
not deploy a proposed virtual helpline. User training, staff development, and the virtual 
helpline remain unlikely in the near term given present levels of PORTALS staffing. 

• Deploy a geographic information system: The past PORTALS Executive Director Howard 
McGinn indicated to the Department of Education (See 3/25/97 letter from Howard McGinn 
to Christina Dunn, Appendix P-6) that the geographic information system mentioned in the 
proposal was the result of a PORTALS clerical error and the project would not be 
undertaken. 

• Conduct ongoing user-based evaluation and feedback: PORTALS did not hire the local 
evaluator planned for int he grant proposal. PORTALS did not submit an evaluation report 
to the Evaluation Consultant in February 1997 (as did the other grant participants). The 
former Project Systems Programmer did experiment with web log analysis software using 
the PORTALS web page as the result of a January 1997 request of the Evaluation 
Consultant. But the former Project Systems Programmer found the software cumbersome to 
deploy and of minimal value in assessing how people use the PORTALS web page. The 
former Project Systems Programmer created a listserv related to the grant implementation 
in March 1996 but it did receive much use. The PORTALS Director of Network Information, 
included a "mailto" webmaster option3 in the PORTALS web page which receives 4-5 
messages a week on average. With these exceptions, the evaluators are aware of no other 
efforts by PORTALS to conduct local, ongoing, user-based evaluation and feedback of its 
programmatic portion of the grant. User-based evaluation and feedback efforts remain 
unlikely in the near term given present levels of PORTALS staffing. 

These grant activities are in addition to normal PORTALS operations and PORTALS overall 

administration of the HEA IIB grant not examined by the evaluators here. 



3 The user of the PORTALS web page will see: PORTALS Webmaster underlined at the bottom of the opening 
page. To the network savvy, this indicates that if the user wants to send a message to the person in charge of the 
website (the webmaster) the user can click on the underlined portion and, assuming the user's web browser is 
configured properly, send a mail message to the person in charge. A significant problem with the mailto option is that it 
does not work from public terminals or for users who do not have e-mail accounts because a return e-maifis required. 
While there are ways around this requirement, they are generally beyond most users. 
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Summary of Evaluator's Activities 



The Onsite Evaluator received several briefings by the Evaluation Consultant prior to the 
onsite visit. The Onsite Evaluator examined the Evaluation Consultant’s reports and pertinent 
correspondence. The Onsite Evaluator examined all available documentation on the project including 
the original grant proposal. The Onsite Evaluator examined the PORTALS web page and other related 
Internet resources. As a result of a preliminary evaluation of the PORTALS website, the Onsite 
Evaluator provided the PORTALS webmaster with an extensive list (in ready to use HTML code 
format) of sources to improve the local, county, state, federal, and international sub-sections of the 
government information section of the PORTALS website. The Onsite Evaluator examined the 
PORTALS web site on May 1, 1997 and provided recommendations for next steps (see Appendix P-1). 

The Onsite Evaluator prepared a preliminary draft report indicating what was tentatively known and 
questions to be pursued during his onsite visit. 

The Onsite Evaluator conducted his onsite visit June 3 to June 26, 1997. The evaluator met with 
the PORTALS interim Executive Director on June 3 and 25, 1997 using pre-structured interview protocols, 
and informally on other occasions. The Onsite Evaluator made a presentation to the PORTALS Council 
of Librarians at Pacific University, Forest Grove Oregon on June 4, 1997 regarding his onsite evaluation. 
On June 9, 1997 the Onsite Evaluator interviewed via telephone the former Project Systems Programmer, 
using pre-structured interview protocols. The evaluator interviewed the Director of Network 
Information, on June 12, 1997 using pre-structured interview protocols and informally on other occasions. 
The Onsite Evaluator collected follow up data via e-mail as needed. 



Chronological Summary of Key Events and Accomplishments 

Summarized, in chronological fashion below, are key activities, milestones, outcomes, and 
accomplishments including principal work products to date given the project's objectives. Omitted are 
regular, ongoing, phone conversations or meetings at various conferences between the Evaluation 
Consultant and PORTALS staff or project participants. 



7/21 / 93 PORTALS founded 

Fall 1993 Dennis Gilbert hired as Project Systems Programmer 

1994-97 Project Systems Programmer develops and tests prototype of PORTALS backbone network 
Summer 

1995 PORTALS web page begun <http://www.portals.org/> 

8/95 Howard McGinn begins as PORTALS Executive Director 

10/1/95 HEA II-B grant begins 

10/95 Millard Johnson, previous Director of Network Development, leaves. 

10/30/95 Presentation on the HEA II-B grant to the Fall Coalition for Networked Information 
meeting in Portland. 

11/1/95 Charles McClure, Evaluation Consultant, meets grant principals in Portland. 
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12/19/95 

12/28/95 

1/22/96 

3/96 



Charles McClure, evaluation consultant conducts conference call with grant principals. 
Purchase of 14 Sun Sparc 5 stations for backbone network. 

Purchase of 14 Pentium microcomputers for backbone network. 

Project Systems Programmer creates a listserv related to the grant implementation which 
did not receive much use. 



Summer 1996 

to 5/97 Project Systems Programmer actively involved in loading and maintaining PORTALS 

licensed databases. This requires more staff time than expected, indeed it is a full time job 
by itself. 

8/96 Purchase and installation of Sun Ultra 3000e (was to be Sun Sparc lOOOe and CD-ROM 

Jukebox but better technology was available, change approved by Christina Dunn 6/18/96). 

8/96 Charles McClure, Evaluation Consultant and Howard McGinn, Executive Director meet in 

Seattle to review grant status. 

9/1/96 Karen Starr, Director of Network Information, hired. 



9/4-5/96 Department of Education onsite evaluation by Christina Dunn, Neil Kaske, and Shirley 
Steele, and visits by other similar project participants from Louisiana, West Virginia, 
Iowa, Colorado, and Maryland. If report filed, PORTALS did not see it. 

10/14/96 Howard McGinn makes LITA presentation on PORTALS/HEA II-B grant. 

1996-1997 Project Systems Programmer in "spare time" makes continuous improvements to prototype 
network in preparation for deployment including: upgrades to cabling, software, network 
security, programming to enable centralized control and repair of physically distributed 
servers, change of domain name to PORTALS, etc. 

Winter 

1996/97 Fiber optic cable linking PORTALS equipment installed by Project Systems Programmer. 

1/23-24/97 Chuck McClure, Evaluation Consultant, makes an onsite visit. Decision made that local 
research assistant cannot be hired. Joe Ryan hired to conduct onsite evaluation. McClure 
requests evaluation reports from grant participants including PORTALS. 

2/97 Project Systems Programmer experiments with web log analysis software. See Web Server 

Statistics for PORTALS. Available: WWW: http://www.portals.org/stats/ 

2/97 Grant participants, excepting PORTALS, file evaluation reports with PORTALS and Dr. 

McClure. 



2/97 Gopher to Lynx conversion begun. 
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2/97 

3/97 

3/31/97 

4/97 

5/16/97 

6/2/97 

6/3/97 

6/4/97 

6/9/97 



6/12/97 

6/25/97 



6/97 

7/97 



Project Systems Programmer develops program to make licensed database tape conversions 
from 9 track tapes. 

PORTALS completes effort to make web page compatible with Lynx, Microsoft Explorer, 
and Netscape browser standards. 

Howard McGinn resigns as PORTALS Executive Director. 

Karen Starr, PORTALS Director of Network Information, adds extensive links to federal, 
state, and local government sources on the Internet. 

Dennis Gilbert, Project Systems Programmer, resigns. 

Jim Kopp begins as PORTALS interim Executive Director. 

Joe Ryan arrives and conducts initial meeting with Jim Kopp. 

PORTALS Council of Librarians meeting. Pacific University, Forest Grove, Oregon. Joe Ryan 
makes presentation regarding his onsite evaluation of the grant. Jim Kopp notes that his is 
a 13 month appointment, that during this period an analysis of the direction and viability 
of PORTALS will be undertaken. 

Joe Ryan interviews via telephone Dennis Gilbert, former Project Systems Programmer. 
Dennis suggests that the backbone network is ready to be deployed with the following 
caveats: allow a week per site for installation and training at member locations, factor in 
post-installation technical support and maintenance of unknown amount (figure at minimum 
visiting each institution one day a month — 25% of a staff person's time per year), 
installation and maintenance would need to be separate from other ongoing duties (such as 
licensed database loading, additional network applications, web master tasks, etc.). 

Joe Ryan interviews Karen Starr, Director of Network Information. 

Joe Ryan interviews Jim Kopp who indicates Board approval for the search for Project 
Systems Programmer replacement, his intent to move toward the use of vendor supported 
(rather than PORTALS supported) licensed databases, the creation of a technology 
committee composed of systems administrators from the PORTALS membership, that 
interviews for a HEA II-B grant principal investigator are underway, and that a user 
services committee met. Jim mentions that the first job of the Project Systems Programmer 
when hired and the newly created technology committee will be to deploy the PORTALS 
backbone network. 

Tom Pfingsten named principal investigator for the grant. 

Lois Cohen hired as Grants Assistant reporting to the Principal Investigator 
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Staff Development Activities 

This sub-section summarizes efforts by grant participants to obtain education and training for 
staff to better implement the grant objectives. PORTALS staff had little time for staff development for 
itself. As far as the onsite evaluator could determine, PORTALS offered no staff development 
activities to the PORTALS membership which relate to the grant. 

Project Work Products 

This sub-section lists and describes the major products, services, and other outcomes as a result of the 
grant. 

PORTALS. Web page. Available: WWW: http://www.portals.org/ 

PORTALS. Web Server Statistics for PORTALS. Available: WWW: http://www.portals.org/stats/ 
Dissemination of Project Results 

This section identifies specific ways grant participants disseminated project findings to the 
local, state, and national communities and the profession including publications, videos, press releases, 
presentations, external training sessions, etc. PORTALS staff made two formal national presentations: 

McGinn, Howard. (1996, October 14). Evaluating library Internet services to citizens. Pittsburgh: ALA. 
LITA/LAMA National Conference. Pittsburgh, PA. 

Ramaley, Judith; Hudson, Kristine; Cohen, Joyce and McGinn, Howard. (1995, October 30). PORTALS. 
Portland, OR: Fall Coalition for Networked Information meeting. 

In addition, the former Executive Director made numerous informal presentations around the country. 
PORTALS staff and others mentioned aspects of the project at various state meetings including the 
Oregon Library Association toward the middle and end of this grant cycle. The evaluators could not 
determine what information the PORTALS membership received about the grant. 

Evaluation Efforts 

This section identifies specific ways PORTALS used to evaluate the project. PORTALS did not 
hire the local evaluator proposed in the grant for reasons not known to the evaluators. PORTALS did 
not file an evaluation report with the Evaluation Consultant in February 1997 as did the other grant 
participants. The former Project Systems Programmer experimented with web log analysis software in 
February 1997, see < http://www.portals.org/stats/>. The PORTALS webmaster made provision for a 
mailto" the webmaster option enabling some feedback from users of the website generating a few (4-5) 
messages per week . The evaluators are not aware of any other formal efforts at evaluation of the grant 
other than that engaged in by the present evaluation team. 

Lessons Learned 

This sub-section identifies significant lessons learned by PORTALS which may also be of 
interest to similar organizations in other settings. 
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PORTALS Is Chronically Under-staffed 

A principal finding is that PORTALS failed to achieve its grant objectives due to a chronic lack 

of staff: 

• PORTALS did not hire staff identified (and funded) in the grant proposal (including the 
local evaluation research assistant). 

• PORTALS staff assigned to the project had other, more compelling responsibilities leaving 
little if any time for grant activities. 

• Staff turnover, and position vacancies significantly added to staffing problems. 

The existing PORTALS staff,, heroically at times attempted to fill the void but these efforts could not 
compensate for the inadequate number of staff. All too often, the PORTALS staff operated due to these 
circumstances in crisis mode with attention to any one detail episodic at best. 

It is outside the scope of this study to determine why the staff shortage exists. Recent 
PORTALS actions begin to address some of these concerns including: 

• Search for a replacement for the Project Systems Programmer is underway 

• Move to vendor support of PORTALS licensed databases, and 

• Outsourcing of the HEA II-B grant's administration (to the Portland State University 
Library Director and a newly hired grant assistant reporting to him). 

Time will tell if the solution found for the PORTALS HEA II-B grant administration is adequate and 
workable. The evaluators remain concerned that PORTALS lacks the staff necessary to complete the 
installation of the backbone network while conducting normal PORTALS duties. 

PORTALS Members Uninvolved in Grant Activities to Date 

The success of the grant required an adequate core PORTALS staff and active participation by 
volunteer staff (most often formed into committees) from the PORTALS membership not otherwise 
participating in the grant. The present grant period saw inadequate PORTALS core staff and little 
participation by the PORTALS members who were not direct grant recipients. Indeed, even staff from 
the PORTALS membership specifically committed in the original grant proposal to achieving certain 
PORTALS component grant objectives did not to help achieve grant objectives. The reasons for the lack 
of PORTALS members participation are not known to the evaluators and outside the scope of the 
evaluation. However the impact of the inadequate PORTALS core staff and the lack of participation 
by the PORTALS members in grant activities is plain: the backbone network is not deployed, provision 
of government information on the PORTALS website is at first delayed and its future uncertain, and 
staff training, continuing education, and dissemination of grant results is negligible. 

An early success of the interim Executive Director is his efforts to enlist voluntary contributions 
of the PORTALS membership's staff time in achieving the objectives agreed to by the PORTALS 
membership. The evaluators hope these efforts will enable the completion of such grant objectives as 
the establishment of the PORTALS backbone network. 
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PORTALS Backbone Network Not Deployed 

The development and deployment of a backbone network among the PORTALS members was to 
be the centerpiece of the PORTALS portion of the grant. To date the deployment of the network is 
incomplete. The former Project Systems Programmer had more than a full time job ordering and testing 
equipment, mounting and maintaining PORTALS licensed databases, and creating and maintaining the 
PORTALS server and web page -- not to mention a range of other activities which included informal 
work for Portland State University and technical troubleshooting for virtually all of the PORTALS 
membership. Despite this, the former Project Systems Programmer found time to do the planning, 
acquisition of equipment and software, and software development for the proposed PORTALS backbone 
network apparently bringing this aspect of the task to the ready to deploy stage. 

Federal Government Information Accessibility Only Minimally Addressed 

PORTALS was to take the lead in making federal government information, with emphasis 
given to resources of interest to Oregon and the Northwest, accessible to PORTALS members. The other 
grant participants were to focus on making state and local information available. The grant proposal 
citing one of the present authors (Ryan, 1994), noted that "Government information is widely available 
on the Internet but is poorly organized, of uneven quality, and difficult for most users to access." 
PORTALS, however, was unable to address this grant component successfully due to a number of factors: 

• Proposed was a website to include federal and other government information. By early 1997 
a PORTALS website existed but without significant government information on it. The 
Evaluation Consultant recommended that the PORTALS Director of Network Information 
to step in an take responsibility for this portion of the PORTALS grant project. The 
Director of Network Information, assisted in part by the Onsite Evaluator prior to his 
onsite visit, created a basic set of links to federal, state, local, and PORTALS member 
government information efforts as a sub-section of the PORTALS website. The continued 
update and maintenance of this section of the PORTALS web page is in question due to the 
more pressing obligations of the PORTALS Director of Network Information. 

• Proposed was a plan to provide access via PORTALS Internet web page to a range of federal 
government databases mounted locally including Census data; National Trade Data Bank; 
National Environmental, Sociological, and Economic Data Bank; and the Regional 
Economic Information System. This activity did not occur and notification sent in a letter 
from the then Executive Director Howard McGinn to Christina Dunn of the Department of 
Education March 25, 1997 (see Appendix P-6). 

• User feedback and involvement were to be critical to the PORTALS effort to make federal 
government information available to the PORTALS membership. An advisory committee 
was to assist PORTALS in the provision of federal and other government information on the 
PORTALS website. Apparently PORTALS convened an advisory committee in late 1995 
and/or early 1996. After some controversy (the evaluators so not know the particulars), the 
advisory committee disbanded. Since then, no PORTALS members or members' users 
participated in this component of the PORTALS portion of the grant. 

The government information portion of the PORTALS web page does not solicit feedback nor 
does PORTALS make provision to receive or process any other feedback regarding 
government information from PORTALS members or users. 
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Despite being a federally funded grant to improve "citizen access to government and other information/' 
the grant made little contribution to improved access to federal government information for PORTALS 
members and their users. 

User Training and Staff Development Not Done 

During the period of the grant, Internet use, specifically using the Internet to access government 
information, went from being uncommon to commonplace among the PORTALS membership. However 
PORTALS played little if any role in supporting its members in the transition to use of the Internet to 
access government information. Promised user support, training, staff development, helplines all did 
not appear. This was an extraordinary opportunity missed -- not for lack of interest or ability but due to 
lack of staff. 

Continuous Ongoing Evaluation Not a Part of PORTALS Practice 

Troubling is the lack of a plan to actively seek out user feedback to PORTALS initiatives, 
specifically the government information provided by itself and its members as part of the grant. There 
appears to be no policy or procedures in place to seek or obtain user feedback, or channel user feedback to 
the appropriate PORTALS member, PORTALS committee, or PORTALS staff member, specifically, in 
this case, as regards the effectiveness of the PORTALS portion of the grant. There appears to be no 
policy in place for ensuring prompt consideration of any user feedback obtained. In sum, internal 
planning for continuous user-based evaluation, the conduct of such evaluation, and policies and 
procedures for using evaluative feedback do not seem to be in place. 

The interim Executive Director points out that it is PORTALS policy that PORTALS (as an 
organization) provides services to its members which, in turn, provide service to users. This is a 
reasonable policy and certainly politically wise. However, this policy puts a potential barrier layer 
of bureaucracy between a service provider (PORTALS) eager to discover if its service is effective and an 
end user eager to offer a suggestion for improvement. The networked environment of the Internet may 
add an additional barrier to obtaining user feedback. In a networked environment with service 
provided by a consortia, who the service provider is may not be at all clear to an end user (indeed, they 
may not even care). The most effective common solution to reducing these potential barriers is a policy 
which clearly articulates role, procedures, and process as it relates to continuous evaluation and 
solicitation of user feedback coupled with training sessions for key public service personnel. The policy 
and subsequent training should make clear who will solicit user feedback and how; how will feedback 
be processed with specific roles identified for PORTALS, PORTALS members, and users; and, who is 
accountable for what aspects of evaluation outcomes. At present, such policy does not exist at 
PORTALS. Without such a policy and training program PORTALS will not be able to determine the 
effectiveness of its grant activities. 

Next Steps 

This sub-section suggests next steps to be taken as this grant period comes to an end and to 
advance the project's objectives after the funding period. 

The Onsite Evaluator arrived the day after the new interim Executive Director, Jim Kopp, 
started. He is currently reviewing the direction, approach, and pace of PORTALS' activities. The 
question of what next steps for PORTALS to take is under active consideration at the moment. The next 
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steps, highlighted below, relate specifically to the grant objectives within the emerging framework 
being articulated by the interim Executive Director and PORTALS membership. 

Deploy the PORTALS Backbone Network 

The original deployment plan proposed in the grant assumes that the equipment is physically 
distributed to each member’s site with much of the day-to-day maintenance and the production and 
management of content on the computers controlled from PORTALS headquarters. The next steps in this 
process are to install the Sun and Pentium workstations at each members' site and train local personnel 
(estimated time one week per site), test and debug the resulting installation, provide ongoing technical 
support (estimated at minimum as one day per month per site — 25% of a technical staff person’s time). 
Estimates are from the former Project Systems Programmer. A potential problem with the present plan 
is that while the equipment is physically housed at member institutions they cannot directly add or 
control the content on the computers. There are at least three other deployment options under 
consideration given the advent of the Internet, a factor not considered in the original grant proposal. 

A second option, is to not physically distribute the equipment, simply link the equipment 
locally at PORTALS headquarters and then to the Northwest Net T1 connection and all PORTALS 
members would have the same access via the Internet as planned now. This would eliminate the need to 
travel to the fourteen sites to install equipment, train staff at each location, and maintain equipment at 
each of the different sites. 

A third option is to simply distribute the equipment to each member with each member 
controlling what is put on the machines and their use and maintenance. The advantage is that the 
deployment is fast, members are responsible for equipment maintenance, and members can use the 
equipment to mount their own information. The computers would be connected to the Internet so that 
information and communication could be shared among other PORTALS members and the wider Internet 
community. The former Project Systems Programmer has concerns about some members ability to install 
and maintain the equipment. The issue of who controls what is placed on the computers and how this 
approach would meet the terms of the grant must be addressed. 

A fourth option assumes deployment as originally planned with policies and procedures in 
place which identifies the network’s purpose and use, how content can be contributed from PORTALS 
members (or even individuals within PORTALS institutions), who would maintain the equipment under 
what conditions, and other governance and standards issues. This mechanism would be developed by a 
PORTALS ’’content committee" with advise from technology experts within the membership. 

There may be other options as well. The hiring of a replacement for the former Project Systems 
Programmer is underway. The interim Executive Director is in the process of constituting a technology 
committee composed of systems administrators from the member institutions to investigate deployment 
options and aid the eventual deployment. 

Populate the PORTALS Server and Network with PORTALS Members Content 

At present, PORTALS struggles to maintain the availability of the licensed databases and a 
very modest web page. A plan to move to vendor support of the licensed databases is set to begin. 

Should the backbone network be deployed, or even with the existing PORTALS website, what content 
should be mounted, how, by whom, for how long, etc? 
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Make Provision for PORTALS Members User Training, Contribute to Staff Development 

The new draft strategic plan (Available: WWW: http://www.portals.org/plantoplan.html) 
by the interim Executive Director targets continuing education for the PORTALS membership as an area 
for attention. Will the focus of training be broadened to areas more germane to the grant's activities. 
How will continuing education needs be identified, by who, how managed and funded? Can meaningful 
staff training be delivered via the backbone network and the Internet? 

Recommendations 

The evaluators offer the following recommendations based on the onsite visit and subsequent 
discussions. 

Deploy Backbone Network 

The backbone network could and should be at the heart of a revitalized PORTALS, offering the 
information products and services that are the tangible benefits of PORTALS membership, and enabling 
efficient communication for administration, planning, and continuing education. The interim Executive 
Director assured the Onsite Evaluator of his commitment to deploy the PORTALS backbone network, 
but how and when remain to be determined. The interim Executive Director and Board took initial 
positive steps by seeking to hire a new Project Systems Programmer and constituting a technology 
committee. 

The evaluators recommend that the interim Executive Director publicly set a date, with 
accompanying plan (addressing some of the issues raised next), when the PORTALS backbone network 
will be in place. Further, the evaluators recommend that PORTALS adopt a network architecture and 
deployment plan which fosters the production and use of the type of scholarly content and 
communication the PORTALS membership values most. 

Address PORTALS Network Content and Communication Policies and Procedures 

The evaluators caution that the technological deployment of the PORTALS network, while a 
significant challenge, may not be the most crucial issue. What content is available on the network? 

Who can contribute content to the network, how, and under what conditions? How and who determines 
these and other governance issues? Do PORTALS members, including the librarians and faculty 
members (and even students) who produce content, believe that they are empowered collaborators in 
the PORTALS network cooperative enterprise? 

Equally important will be fostering communication on the PORTALS network about PORTALS 
specific issues and about issues of interest to the PORTALS scholarly community. How will this be 
accomplished? What network vehicles will be in place to encourage communication among PORTALS 
members about PORTALS or stimulate the scholarly pursuits of PORTALS member scholars? Who will 
control these communication vehicles? Will PORTALS offer listservs or chat rooms, on what topics, 
and managed by whom? Will student scholars be offered PORTALS space to mount web pages, under 
what conditions, and who will manage the program? What will be the first continuing education course 
offered on the PORTALS network? How will it be offered? The evaluators suggest that it is past time 
for PORTALS to consider these and other related issues regarding the use of the PORTALS network for 
communication by PORTALS members. 
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The evaluators caution, that the database licensing issues, no matter how visible, expensive, 
and rancorous, may not represent the future for the PORTALS membership. The content that may 
matter most to PORTALS members and the larger education community may be the content that is 
locally produced and given a home on the PORTALS network. The communication that may matter 
most may be the communication on local issues raised and resolved using the PORTALS network. But 
without policies and procedures for how content, including licensed databases, is to be presented and 
communication to occur, the network will languish. 

The evaluators recommend a committee to develop policies and procedures which ensure that 
all PORTALS members can contribute to the educational asset the backbone network represents. It is 
possible to construe these matters to be in the domain of PORTALs internal management and thus 
outside the scope of this report. The evaluators caution however, that a principal contributing cause to 
PORTALs failure to meet its grant objectives to date is PORTALS lack of a policy mechanism to 
address what government information content will be made accessible on the backbone network, who 
will create, select, enter, maintain, deselect such content and how, etc. Without PORTALS network 
content and communication policies and procedures, grant objectives will not be met or met well. 

Make Federal and other Government Information Accessible to PORTALS Members 

PORTALS was to make federal government information accessible on its network as part of the 
grant. The evaluators have two concerns. First, we believe that government information should be a 
key component of the content offered by the PORTALS network. Second, we do not believe the day-to- 
day ongoing update and expansion of the government information section should be the responsibility of 
the Director of Network Information. The Director of Network Information stepped in to solve an 
immediate problem and did a competent job, but she may now have other more pressing responsibilities. 
The ongoing responsibility for the provision of federal and other government information belongs 
within the PORTALS membership at the subject expert level. At this late date, PORTALS might best 
contribute by serving as catalyst and coordinator of the efforts of its members’ experts. 

The evaluators recommend a committee be formed to coordinate the development and 
maintenance of the government resource portion of the PORTALS network. Logical committee members 
include the government documents and law librarians from member institutions. However we suggest 
PORTALS consider inviting representatives from the grant participants (Multnomah County Library, 
State Library, and Oregon Historical Society) who produce government information as well as other 
local government webmasters (such as the City of Portland) to participate. The dual charge to the 
committee should be on the one hand to determine the government information needs of PORTALS 
members to be best met via the network and PORTALS, and on the other, to determine what government 
information can PORTALS members uniquely contribute to the PORTALS network. For PORTALS 
members the situation is much as Ryan portrayed it in 1994, "Government information is widely 
available on the Internet but is poorly organized, of uneven quality, and difficult for most users to 
access." With a coordinated effort led by PORTALS, this situation does not have to be the same today 
for its members. 

Address PORTALS Members’ Staff and User Training Needs 

If the PORTALS network is to make available a rich array of digital content who will alert 
staff and users to its availability, who will train staff and users to taken advantage of these sources? 
Should fourteen essentially identical guides to using each licensed database have to be produced? Is it 
possible to use the PORTALS network to deliver continuing education to staff? Is it possible to use the 
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PORTALS network to deliver courses produced collaboratively by PORTALS faculty members to 
students and others at PORTALS institutions? 

The evaluators note the new PORTALS strategic plan's emphasis on staff development and 
continuing education. The evaluators suggest a modest broadening of the scope of the proposed staff 
development effort to include other needs of the PORTALs membership including those related to the 
grant. Specifically the evaluators recommend a committee to develop a plan for (1) a PORTALS guide 
to each licensed government database (e.g., Medline and ERIC) suitable for all locations to use - the 
guides might be mounted on the PORTALS network to solve the publication cost question (2) Create a 
staff development program of interest to the PORTALS membership delivered in whole or in part on 
the PORTALS network (3) Develop a course to be delivered in whole or in part over the network - an 
introduction to government resources on the Internet for researchers might be a place to start, and with 
these accomplished: (4) Create a structure, policy, and mechanism to enable other staff development 
needs to be identified and met by PORTALS. These tasks ought to contribute to establishing a 
knowledge base and capacity to enable PORTALS to decide what is feasible and desirable in this area. 

Embed Evaluation Mechanism 

A PORTALS headquarters staff out of touch with the PORTALS membership, not only the 
Board and Council of Librarians, but the middle managers, front line staff, and ultimately the users of 
each institutional member, can be fatal to PORTALS. Without a systematic plan to interact with and 
collect data from these key stakeholder groups PORTALS headquarters will not know whether they 
are meeting grant objectives or members needs. Furthermore, the PORTALS membership will believe 
that PORTALS headquarters does not care what the members think. The statement at the June 4 
Council of Librarians meeting by a Council member that the Onsite Evaluator’s presentation was the 
first time the Council had heard a clear explanation of what the grant was about was a clear indicator 
of the need. The grant, with its explicit requirements for continuous user-based evaluation, represented 
an opportunity to tackle the evaluation problem and get it right. The evaluation problem remains. 

The evaluators recommend that PORTALS evaluate locally and communicate findings widely. 
Specifically a local evaluator should be hired to systematically "take the pulse” of the PORTALS 
membership. How well are various products, services, and other initiatives being received? How could 
these products, services, and initiatives be improved? What are new directions PORTALS might take, 
generated from within the PORTALS membership and outside it? Specific attention should be given to 
the end user’s evaluation of PORTALS information resources and services. The evaluators recommend a 
committee be created to work in conjunction with the local evaluator. The charge to the committee 
would be to develop an evaluation plan with specified targets and commitments, involving PORTALS 
members where possible, and assigning PORTALS staff certain responsibilities. The local evaluator 
and evaluation committee should give specific attention to training other PORTALS committees in 
appropriate evaluation practices and in monitoring evaluation efforts of those committees. The 
evaluation committee should develop a mechanism and plan for disseminating evaluation results (using 
the network where appropriate) to the PORTALS membership and, where appropriate, interested 
outside professionals. The evaluators recommend that the local evaluator's work and the evaluation 
committee’s work be public and that accountability mechanisms be in place. 

The evaluators suggest that interactive forms specifically soliciting feedback be added to the 
PORTALS website. A good example is at MCL’s RITNet <http://www.region.portland.or.us/>. Many 
users may not recognize the availability of the existing "mailto” feature. Others will not know that 
their comments are welcome. Still others will not respond unless specifically invited. 
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Hire Necessary PORTALS Staff, Involve the PORTALS Membership 

A consortium like PORTALS depends upon a critical mass of dedicated core staff and the 
essential voluntary staff participation of consortium members: neither is enough to get the job done, 
both are necessary. The PORTALS office was and is under-staffed. The evaluators note with approval 
the interim Executive Director and Board’s action to hire a new Project Systems Programmer. The 
outsourcing of PORTALS licensed database activity and the administration of the HEA II-B grant may 
also free PORTALS core staff time. The evaluators question whether these personnel acquisitions and 
changes are enough. Also needed are: 

• A dedicated administrative assistant hired by and reporting to the Executive Director. 

This person would handle the ’’parking, pastries, and paperwork” which can fatally mire 
a multi-jurisdictional consortium. Due to the confidential nature of the job, handling 
sensitive information from all of the consortium members, careful attention to how the 
position is funded and the reporting relationship is needed. This position should report 
directly to the Executive Director not through a member institution to him. 

• A PORTALS-based local evaluator, originally designated in the original grant proposal, to 
systematically tackle the evaluation issues discussed above. 

• A program administrator, or more than one, to manage the day-today pragmatics of the 
number of initiatives already underway (e.g., database licensing, cooperative collection 
development, reciprocal borrowing) and the future initiatives to follow. There is need for 
someone to "ride herd" on the day-to-day process of making these initiatives happen 
freeing the Executive Director to negotiate and make the tough decisions necessary. 

• A public relations person to get the PORTALS message out to members, stakeholders, to the 
region, to the profession, and to coordinate the development of future grant proposals is 
essential. In a consortium, communication is critical, should it be left to chance? 

• A web master to coordinate the management and production of content and communication on 
the PORTALS network as discussed above. This person would administer policy and 
maintain standards, pro-actively identify and create content, initiate and participate in 
web related projects undertaken by PORTALS members to be mounted on the PORTALS site, 
and serve as a resource to the PORTALS membership. 

And there may be other positions needed including a user services /staff development officer, and a full 
time grants administrator. But an adequate complement of PORTALS office staff will not be enough 
without a committed PORTALS membership. 

There is a need to mobilize the staff of the PORTALS membership to contribute their time to a 
range of worthy ends associated with the present grant, future phases of the HEA II-B grant, and future 
grants. Sometimes a consortium is not lead so much as energy is unleashed and appropriately directed. 
The evaluators note with approval the interim Executive Director’s efforts to re-energize PORTALS 
committees. But the interim Executive Director and his staff cannot do the job by themselves, 
particularly as under-staffed as they are. Without the active involvement of the PORTALS 
membership, achieving present and future grant objectives are in jeopardy. 
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III. Oregon State Library (OSL) 

This section of the report examines the Oregon State Library programmatic participation in the 
HEA II-B grant including a component designed to provide citizen access to state government 
information and the Project JumpStart (Oregon Internet connectivity grant program) component, these 
components receive separate treatment here. Each project's sub-section includes: program objectives 
using language from the original grant proposal and outcomes to date in brief, the evaluator's activities 
in order to compile each sub-section of the report, a chronological summary of key events and 
accomplishments, staff development activities, project work products, efforts to disseminate project 
results, evaluation efforts, lessons learned, next steps, and the evaluators' recommendations. 



Provide Citizen Access to State Government Information 
Project Objective and Outcomes in Brief 

This sub-section identifies initial project objectives (using text from the grant proposal) for the 
State Library portion of the HEA II-B grant and summarizes progress to date. 

Oregon State Library will deliver important state government information and transaction 
services to Oregon citizens via the Internet. Specifically, OSL will: 

• Provide Citizen Access to State Government Information: including an index to state 
government web sites (using Harvest search engine), and access to Oregon Revised Statutes, 
Oregon Revised Rules, Appellate Court Decisions, and the Oregon Index. OSL will create 
with State Printer an online state document ordering clearinghouse. This system will 
establish a centralized online state document purchasing and ordering system to provide 
state publications from participating agencies. The system will include online browsing of a 
publication inventory and user-friendly and secure ordering and payment mechanisms. 

• Conduct ongoing user-based evaluation/feedback. 

This portion of the State Library's grant activities met with great success. 

The level of knowledge and interest in the provision of Oregon state government by electronic 
means went from low interest to a high priority among Oregon state agencies during the grant period. 
The grant proposal noted the State Library's intent to mount Oregon Revised Statutes, Oregon Revised 
Rules, and Appellate Court Decisions on the State Library's web site. However, agencies other then 
OSL produce each of these data sources. There was initial interest in allowing the State Library to 
mount these sources on its own site. However during the grant period, the individual state agencies 
developed their own ability to mount these sources on their own websites. These other agencies still 
approach the State Library for advice on a consultative basis. The State Library includes links on its 
website to the above mentioned network resources as appropriate. As an alternative, the State Library, 
simply mounted other state government information sources in place of those originally proposed. 

Sources that the State Library did mount using grant funds include: the Oregon State Governor's 
Page, Oregon State Library home page, Oregon Index (current and backfiles 1987 to present), the Oregon 
WorkSmart page (specifically designed for Oregon state employees - a state intranet if you will), and 
most recently the Global Index to Oregon State Government Websites and the Oregon Regional List of 
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Serials (ORULS). The Oregon Index, an index to many of the state’s principal newspapers, is produced 
by the State Library in cooperation with other libraries within the state. 

The original grant also proposed using the Harvest search engine to implement a Global Search 
Engine to index state government web sites. In November 1996, after using substantial consultant time to 
try to create such a global search index, Ernest Perez, the Automated Systems Administrator, 
determined that this freeware package would not do the job. The major reasons for abandoning this 
package was that it was unsupported and required a skilled Unix technical support person which the 
State Library did not have. This delayed the development of the index to state government web sites 
until this year when OSL found "Phantom," a suitable Macintosh-based, search engine application. 

The index is in operation at present. Recently, the state Internet Advisory Committee, the group of the 
state agency webmasters, agreed to cooperate with the project. The state Library will design a simple, 
controlled vocabulary, indexing approach, using the HTML Meta field, to assist with the full-text 
retrieval of agency sources. 

The original grant proposed that the State Library, in cooperation with the State Printer 
would make available an online state document ordering clearinghouse. At present, this project is 
nearing completion. The principal delay is finding a secure payment mechanism. In conversations with 
the State Printer he indicated the target date for public access to this ordering clearinghouse to be 
September 1997. 

In sum, the State Library met or exceeded its targets in its efforts to make State government 
information available to its citizens. The State Library continues to be viewed as a key player and 
resource by other state agencies, even as they become more interested and proficient in using the Internet 
to make government information available to citizens themselves. The State Library, seeking to 
contribute to the state's capacity to provide information via the Internet, hosts most of the internal 
state government listservs, and several listservs for agency/public and agency/media communications. 

Summary of Evaluator's Onsite Activities 

The onsite evaluator reviewed all Oregon State Library (OSL) and related web sites (see 
Appendix P-6). He reviewed all available documentation including the original grant proposal, the 
Evaluation Consultant's reports, and the evaluation report filed by the State Library (Perez, 1997). 

The Onsite Evaluator then prepared a preliminary draft report indicating what was tentatively 
known and questions to be pursued onsite. On June 6, 1997 the Onsite Evaluator met at the State Library 
in Salem, Oregon and interviewed (using pre-structured protocols) the following people: Ernest Perez, 
Automated Systems Administrator; Jim Scheppke, State Librarian; Scott Smith, Information Systems 
Consultant, Strategic Planning & Review, Department of Administrative Services and Mike Freese, 
State Printer; and Roy Tumbugh, State Archivist. The Onsite Evaluator made follow up e-mail 
requests as needed. The OSL key liaisons to the evaluation study checked the Onsite Evaluators 
findings, in the form of a draft report. 

Chronological Summary of Key Events and Accomplishments 

Summarized, in chronological fashion below, are key activities, milestones, outcomes, and 
accomplishments including principal work products to date given the project's objectives. 
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1994/1995 A state sponsored "Leadership Oregon" Project bringing state government managers together 
on a regular basis served as the catalyst for the grant proposal and furthered contacts and 
resources for successfully meeting the grants objectives. The initial Oregon state government 
venture onto the Internet was the "Oregon Online Gopher/* hosted on a Sun LX workstation 
at the State Library. Initial project leaders, drawn together by the Leadership Oregon 
sessions, included: Ernest Perez, State Library Automated Systems Administrator; Mike 
Freese, State Printer; and Scott Smith, Information Systems Coordinator, Strategic 
Planning & Review, DAS (Department of Administrative Services). 

Jim Scheppke, State Librarian, articulates a vision in the Oregon Information Highway 
Project (OIHP) Report. The grant proposal sought to help achieve the goals set out in this 
vision statement. Available: WWW: http://www.osl.state.or.us/orpac/OIHP.html 

PORTALS applies for the U.S. Department of Education HEA II-B grant 

State Library purchases Macintosh with internal funds for Oregon State Library web page. 

The HEA II-B grant begins. 

Purchase and installation of Sun SparcS workstation using grant funds. 

Spring 1996 The State Library applies for copyright permission from the Legislative Joint Committee 
on the Legislative Counsel, copyright owner of the Oregon Revised Statutes (ORS). The 
Joint Committee gives permission to offer unpaid public access to this important resource, to 
both the State Library and to the Office of the Legislative Counsel. This latter 
organization, the actual publisher of the ORS, decides to itself host a version of this 
database on the Internet. Available: WWW: (search engine) 
http://landru.leg.state.or.us/searchl.html Gopher: 

gopher://gopher.leg.state.or.us:70/ll/ors95.dir The State Library homepage provides a 
link to this source, but decided not to duplicate an existing online resource. 

3/96 Oregon State Library web page debut. Available: WWW: 

http://www.osl.state.or.us/oslhome.html Public access items available as of June 1997 
include: State Library online catalog, Library Special Services (including ILL, renewals, 
non-OPAC requests), Census Information, Oregon Document Depository Program, Genealogy 
Information, Access to other Oregon state government databases and information files, 
Grants Information, Oregon Topical Files, "Letter to Libraries Online" library newsletter 
(see references), Talking Book and Braille Services Information, Library Development 
Office information and data files, the libs-or mailing list (statewide library mailing list) 
and ORULS database (Oregon Regional Union List of Serials). 

4/96 Oregon State Governor's web page mounted on Sun Sparc5 funded by PORTALS HEA II-B 

grant. Available: WWW: http://www.govemor.state.or.us/govemor.html 

11/2/96 The State Library notified PORTALS that Harvest is unsatisfactory for providing an index 
to state government web sites because of software complexity and lack of support and lack of 
internal expert Unix staff support. This delayed implementation of a global index to 
government websites until mid-1997. 



1/95 

2/6/95 

6/95 

10/1/95 

1/96 
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12/96 

1/97 

1/97 



2/97 

2/97 

4/18/97 

4/97 

6/97 

7/97 



State Library turns off its Oregon Online gopher site in favor of the DAS-operated Oregon 
Online web homepage. 

State Department of Administrative Services mounts Oregon Online. Available: WWW: 
http://www.state.or.us/ This central State website, with links to all state Web and 
gopher servers, reduces pressure to prematurely mount an index to state government 
information. The search for an appropriate search engine continues. 

Current version (1995 to present) of the Oregon Index made available on the web by the 
State Library using a WindowsNT/ O’Reilly server with Inmagic search engine purchased 
from grant funds. Available: WWW: http://www.osl.state.or.us/orpac/orindhome.html 
The Oregon Index is an online index to several Oregon newspapers (including: Coos Bay 
World 1996 to present, Corvallis Gazette Times 1988 to present, Klamath Falls Herald & 
News 1996 to present, LaGrande Observer 1989 to present, Lake Oswego Express (weekly) 
1996 to present, Salem Statesman Journal 1987 to present, Tigard Times 1996 to present) and 
to magazine articles about Oregon cooperatively indexed by participating libraries and 
coordinated and made available on a website by the State Library. The Index started in 
1987 using Personal Librarian software. The index later migrated to BRS search software, 
and finally to the present Inmagic WebPublisher package. The total index span now 
contains approximately 400,000 records. 

Backfiles (1987 - 1994) of the Oregon Index made accessible on the Internet by the State 
Library. Available: WWW: http://db.osl.state.or.us/orind2.htm 

The Oregon Judicial Department begins planning to mount the State Appellate Court 
decisions itself, no target date set. The State Library planned to mount this source on its 
website but decides not to duplicate an existing network resource. 

The Secretary of State, Archives Department mounts the Oregon Administrative Rules. 
Available: WWW: http://arcweb.sos.state.or.us/oarsos.html The State Library 
originally intended to mount this source on its site, but again decided to let the original 
data publisher take the lead in providing public access. The State Library homepage does 
provides a link to this government information resource. 

Prototype Global Index to Oregon State Government Web Sites" using Macintosh-based 
"Phantom” software (purchased by grant) installed. Available: WWW: 
http:/ /index, osl. state, or. us:8080 

The "Global Index to Oregon State Government Web Sites" moves to public access status. 
Publicity includes an announcement on the libs-or listserv reaching virtually all the 
libraries of the state, an e-mail announcement to all major Oregon Internet Service 
Providers, and announcements on Ed-Net/Compass, the state telecommunications and 
video-conferencing network. 

State Library plans to utilize the under used Sun SPARC5 workstation purchased as part of 
the grant as a host Web site for smaller state agencies, commissions, and boards, who do not 
have the infrastructure or technical expertise needed to host their own Web sites. 

Prototype pages for this activity will be the Advocacy Commissions (handling 
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diversity /advocacy activities for minority and women's groups) and the Pioneer 
Cemeteries Commission. OSL plans to provide basic technical support for the Web site, and 
basic-level HTML technical help as needed. 

9/97 State document ordering clearinghouse targeted to be in operation. The State Printer in 

cooperation with other agencies including the State Library will establish a centralized 
online state document purchasing and ordering systems to provide state publications from 
participating agencies. System will include online browsing of publication inventory and 
user-friendly and secure ordering and payment mechanisms. As of 6/97 equipment and 
software purchased. 



Staff Development Activities 

This section summarizes efforts by grant participants to obtain education and training for staff 
to better implement the grant objectives. 

A state sponsored Leadership Oregon Project in 1994/1995 bringing state government managers 
together on a regular basis served as the catalyst for the grant proposal and provided the contacts and 
resources for successfully meeting the grant's objectives. OSL staff use local state government listservs 
(for which the State Library provides space) along with some national listservs (web41ibs, pacs-1). 
The State Library did not find courses and conferences to be of use with the exception of C-PUG , a state 
electronic publishers user quarterly meeting. 



Project Work Products 

This sub-section lists and describes the major products, services, and other outcomes as a result of 
the grant: 

Oregon State Library home page. Available: WWW: http://www.osl.state.or.us/oslhome.html 

Global Search of Oregon State Agency Web Sites. Available: WWW: http://index.osl.state.or.us:8080 

Oregon State Library's Global Search of Oregon State Agency Web Sites indexes the following agencies 
as of July 3, 1997: 

Adult and Family Services Division (5/2/97 - 124), Board of Barbers and hairdressers (6/16/97 - 65), Criminal 
Justice Services Division (6/25/97 - 10), Department of Human Resources (5/1/97 - 540), Department of Land 
Conservation and Development (5/1/97 - 240), Employment Department (6/11/97 - 537), Information Resources 
Management Division (7/2/97 - 250), Insurance Pool Governing Board (6/20/97 - 14), Law Enforcement Data 
Systems (5/1/97 - 7), Legislative Administration (6/18/97 - 498), Office of Alcohol and Drug Abuse Programs 
(5/2/97 - 1), Office of Economic Analysis (6/26/97 - 79), Office of Educational Policy and Planning (6/20/97 - 85), 
Office of Emergency Management (6/27/97 - 82), Office of Energy (6/27/97 - 201), Office of the Oregon Health Plan 
Administrator (7/1/97 - 146), Oregon Board of Accountancy (6/15/97 - 58), Oregon Board of Chiropractic 
Examiners (5/2/97 - 16), Oregon Board of Medical Examiners (6/20/97 - 10), Oregon Commission for the Blind 
(5/2/97 - 99), Oregon Commission on Children and Families (6/18/97 - 9), Oregon Department of Administrative 
Services (6/21 /9 7 - 2125), Oregon Department of Agriculture (4/23/97 - 726), Oregon Department of Consumer & 
Business Services (6/9/97 - 2/65), Oregon Department of Corrections (5/1/97 - 163), Oregon Department of 
Environmental Quality (4/30/97 - 508), Oregon Department of Fish and Wildlife (6/11/97 - 526), Oregon 
Department of Forestry (6/11/97- 2499), Oregon Department of Geology and Mineral Industries’ (DOG AMI) 

(6/12/9 7 - 127), Oregon Department of Justice (5/1/97 - 50), Oregon Department of Revenue (6/13/97 - 63), Oregon 
Department of Transportation (6/14/97 - 1022), Oregon Department ofVeteran s Affairs (5/2/97 - 9), Oregon 
Dispute Resolution Commission (6/19/97 - 42), Oregon Economic Development Department (6/10/97 - 990), Oregon 
ED-NET (6/25/97 - 366), Oregon Governor John Kitzhaber (6/3/97 - 194), Oregon Hazelnut Commission (6/19/97 
- 37), Oregon Health Division (5/1/97 - 243), Oregon Health Division Licensing Programs (5/2/97 - 65), Oregon 
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Housing and Community Services (6/12/97 - 23), Oregon Judicial Department Publications ( 7 / 1/97 - 27), Oregon 
Lottery (6/13/97 - 319), Oregon Oil Heat Commission (6/20/97 - 89), Oregon Online (5/2/97 - 2764), Oregon Public 
Utility Commission (6/20/97 - 432), Oregon Racing Commission (6/22/97 - 70), Oregon State Fair & Exposition 
Center (6/11/97 - 12), Oregon State Library (4/22/97 - 1747), Oregon State Marine Board (5/1/97 - 100), Oregon 
State Scholarship Commission (6/23/97 - 147), Oregon State Treasury (6/14/97 - 33), Oregon Youth Authority 
(6/15/97 - 67), Public Employee Retirement System (6/13/97 - 87), Senior and Disabled Services Division (5/2/97 - 
154), Services for Children and Families (5/2/97 - 82), Technology Learning Center (5/1/97 - 63), 
Telecommunications (5/1/97 - 12), Water Resources Department (6/14/97 - 203), Willamette Valley Livability 
Forum (6/24/97 - 445) 

Oregon Index. (1995 to present). Available: WWW: http://www.osl.state.or.us/orpac/orindhome.html 

Oregon Index Backfile. (1987 - 1994). Available: WWW: http://db.osl.state.or.us/orind2.htm 

Oregon Regional List of Serials (ORULS). Available: WWW: 
http://www.osl.state.or.us/oslhome/orulshome.html 

Oregon State Governor's web page. Available: WWW: 
http://www.govemor.state.or.us/govemor.html 

WorkSmart. Available: WWW: http://www.state.or.us/worksmrt.htm 
Dissemination of Project Results 

This section identifies specific ways grant participants disseminated project findings to the 
local, state, and national communities and the profession including publications, videos, press releases, 
presentations, external training sessions, etc. 

There was active dissemination of the project efforts within Oregon State government 
primarily through a state government web masters group. However there is little dissemination of the 
project’s efforts outside the state, except through e-mail and listservs (i.e. libs-or). Additional 
dissemination took place at meetings at the Oregon Library Association and Oregon Educational Media 
Association annual conferences (mostly about the Oregon Information Highway project and the Oregon 
Index). Ernest Perez is active in teaching HTML and web design in Oregon including classes at 
Marylhurst College, Oregon Health Sciences University, and Western Oregon State College, some of 
his thoughts are summarized on the HyperLibrarian Thoughts web site (Available: WWW: 
h ttp : / / w w w . open . org / pereze / index .htm ) . 

Evaluation Efforts 

This section identifies specific ways the State Library used to evaluate the project. Ernest Perez (1997, 
February) submitted a grant evaluation report requested by the Evaluation Consultant to PORTALS. In 
addition, he tried various log analysis software but is, at present, dissatisfied with their ease of use 
and what they credibly reveal about web site users. The State Library used no additional evaluative 
mechanisms beyond existing State Library practice. 

Lessons Learned 

This sub-section identifies significant lessons learned by the State Library which may also be 
of interest to similar organizations in other settings. The following lessons learned are from a group 
interview with Ernest Perez, State Library Automated Systems Administrator; Mike Freese, State 
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Printer; and Scott Smith, Information Systems Consultant, Strategic Planning & Review, Department of 
Administrative Services: 

• The development of a web page is one thing, the maintenance of the site is something else, 
person hours for site maintenance is high. 

• "It is like we are trying to do three jobs at once: keeping up existing services, keeping up 
with technological change, and initiating new programs and services." 

• Partnering with other state agencies and outside government and leveraging resources and 
skills is the norm these days. 

• Hire the highest quality people you can find, look for motivation, and look in unexpected 
places to find them. Then support these people with ongoing training. 

• Find resources where you can. The State Library makes great use of equipment cast off from 
other agencies. 

All of these officials are active in the development of Internet based state government information 
services. 

Next Steps 

This sub-section suggests next steps to be taken as this grant period comes to an end and to 
advance the project s objectives after the funding period. 

The State Library plans to add to the state library web site including: a list of available 
volunteer opportunities in state government agencies (summer), a yellow pages listing of state 
government services to be housed at the state library (late summer), and a web version of the Oregon 
‘ Blue Book (fall). The State Printer plans to announce the State document ordering clearinghouse in 
September. Ernest Perez would like to test the useability of the existing State Library web sites using 
brief focus groups of 3-4 people modeled after a techniques used by Jakob Nielsen (see References below). 

The State Library plans to utilize the under used Sun SPARC5 workstation purchased as part of 
the grant as a host Web site for smaller state agencies, commissions, and boards, who do not have the 
infrastructure or technical expertise needed to host their own Web sites. Initial plans are to provide 
basic technical support for the Web site, and basic-level HTML technical help. 

Recommendations 

The evaluators offer the following recommendations based on the onsite visit and subsequent 
discussions. 

The State Library is active in making state government information accessible to citizens via 
the Internet and informally assisting other state agencies to do the same. The State Library did not 
make available the sources mentioned in the original grant proposal. This is because the originating 
agencies, in a much changed environment, choose to make their own sources available on their own sites. 
Instead, the State Library provided access, technical support, encouragement, and maintenance to other 
state government information including: Oregon State Governor's Page, Oregon State Library page, 
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Oregon Index (current and backfiles 1987 to present), Oregon Regional List of Serials, the Oregon 
WorkSmart page (specifically designed for Oregon state employees - a state intranet), and most 
recently Oregon Regional List of Serials (ORULS). 

At present, the Sun Sparc 5 workstation purchased by the grant is under utilized, only the 
governor's web page is mounted on it. This situation is not unreasonable. Shifts in technology and the 
technical knowledge of the staff made it more efficient to mount the web sites on Windows and MAC 
based microcomputers rather than the Sun workstation. The State Library plans to utilize the 
workstation to provide web pages to smaller units of state government which the evaluators find 
reasonable. The evaluators expect that the State Library will keep the PORTALS interim Executive 
Director apprised of its plans and success and file a written report with him at the end of September 
when the grant period expires. 
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Project JumpStart: The Oregon Internet Connectivity Grant Program 



Project Objective and Outcomes in Brief 

This sub-section identifies initial project objectives (using text from the grant proposal) for the 
State Library JumpStart portion of the HEA II-B grant and summarizes progress to date. 

Oregon State Library will deliver important government state government information and 
transaction services to Oregon citizens via the Internet. Specifically, OSL will: 

• Connect rural libraries to the Internet via Project JumpStart to enhance access to government: 
The "Oregon Internet Connectivity Grant Program' 1 : Rural libraries will receive equipment, 
installation, user-friendly interface, helplines, and training to connect to the Internet. 

• Conduct ongoing user-based evaluation/feedback. 

This portion of the State Library's grant activities also met with great success. 

The Oregon State Library in partnership with PORTALS, the U.S. Department of Education, 
the Oregon Information Highway Project, and the Oregon Independent Telephone Association initiated 
Project JumpStart: The "Oregon Internet Connectivity Grant Program" to connect rural public libraries to 
the Internet. Rural libraries applied to a competitive grant program advertised in the State Library's 
LTLO - Letter to Libraries Online (see References below), and via direct mailing to all public and many 
school libraries. Upon selection a JumpStart library received the following support: 

• Two members from the library (one could be from the library's community) received two 
days of training at an Internet Bootcamp. The training was sub-contracted to Oregon State 
University (OSU), and held in Corvallis Oregon. 

• Hardware and pre-installed software (chosen by the State Library and OSU). Hardware 
included a 486/100 Mhz computer (upgraded to 133 MHz in Jumpstart II) with 8 MB RAM, 
630 MB hard drive, cd-rom drive (in JumpStart II), sound blaster card with earphones, 15 M 
color monitor (upgraded to 17" monitor in JumpStart II), Brother 630 b/w laser jet printer 
and a 28.8k U.S. Robotics modem. Software included: DOS 6.2, Windows 3.1, Trumpet 
Winsock, Netscape 2.0, NCSA Telnet, Adobe Acrobat, and WinPac Z39.50 catalog browser. 
The JumpStart I package was approximately $1,994 (from Proteon in California), JumpStart 
II package was approximately $2,500. The first group received a Z39.50 client subsequently 
dropped for JumpStart II. JumpStart II libraries received a security package. 

• Up to a year of free, dialup connection via an Internet service provider (ISP). OSL selected 
the ISP and negotiated rates for each library. OSL gave funds to each JumpStart I library 
and then the library paid the ISP. While awkward, the funding mechanism gave the 
library a real sense of the costs, influence with the ISP, and a backup contact for technical 
support as well as training, the State Library paid ISPs directly during JumpStart II due to 
budget restrictions. 
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• Several months of ongoing technical support via in-person visit, telephone, and e-mail, 
access to the Oregon Public Libraries web page, and a listserv limited to JumpStart 
participating libraries. OSL sub-contracted with OSU initially(March - December 1996), 
with subsequent rescued support provided by the State Library. 

• The Jumpstart program required that each library have two people involved, one project 
manager and one computer literate person who could be from the community. The Internet 
service must be available to the public. Libraries must locally publicize the availability 
of the service. Libraries must develop a training plan to train the public. Libraries must 
file monthly grant activity reports (see Appendix J-l for an example) and a project report at 
the end of 12 months. Libraries must file a JumpStart Internet Success Story form (see 
Appendix J-2). Libraries must put in wiring and phone line for the project. The State 
Library suggested (but did not require) that each library develop a local library board 
approved acceptable use policy. The ’’JumpStart Project Manager Certification” form (see 
Appendix J-3) summarizes many of these requirements. 

To date 46 rural public and school libraries have Internet connections. In addition, eight additional 
libraries that did not require training or support received infrastructure grants for equipment. In total, 
95% of Oregon’s public libraries now have Internet connections. 

Summary of Evaluator's Onsite Activities 

The Onsite Evaluator engaged in the following activities as part of the evaluation: 

• Reviewed Oregon Public Libraries web site prior to onsite visit. For evaluation summary see 
Appendix P-1. 

• Interviewed (using pre-structured protocols) Rushton Brandis, Network Development 
Consultant with the State Library on June 6, 1997. 

• Visited two JumpStart libraries. On June 13, 1997 visited Seaside Public Library, Seaside 
Oregon (population served: 5,655) and interviewed (using pre-structured protocols) Reita 
Fackerell, Library Director and Paula Clark local JumpStart Project Manager. On June 16, 
1997 visited Driftwood Public Library, Lincoln City Oregon (population served: 10, 937) 
and interviewed (using pre-structured protocols) Susan Jenkins, Assistant Library Director 
and local Project Manager and Yueh-lin Chen, Cataloger and Systems Administrator. Both 
libraries were part of the first JumpStart cohort funded by PORTALS using the grant. 

• Interviewed (using pre-structured protocols) on June 20, 1997, members of the Government 
Information Sharing Project at Valley Library, Oregon State University including: 

Charlene Grasse, Project Manager; Stephen Mosley Research Assistant, Information 
Services (the technical support Consultant); Judy Cross, Government Documents Librarian; 
Cheryl Middleton, Life Sciences Librarian and Kerry Otto. This group assisted in the 
selection and preparation of the hardware and software given to JumpStart libraries, 
provided training at two weekend bootcamps, and ongoing technical support. 

• Examined a variety of documentation onsite and a pre-publication version of Middleton & 
Cross’s (forthcoming) article recounting OSU’s experience with Project JumpStart (a 
manuscript version is included as Appendix J-4). 
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The consultant used a pre-structured interview protocol, taped interviews where possible with 
interviewee agreement and privacy assured, kept additional interviewer notes, and conducted follow- 
up e-mail cross-checks as necessary. The JumpStart projects’ key liaison to the evaluation study 
checked the Onsite Evaluators findings, in the form of a draft report. 

Chronological Summary of Key Events and Accomplishments 

Summarized, in chronological fashion below, are key activities, milestones, outcomes, and 
accomplishments including principal work products to date given the project’s objectives. 

State Library's Experience 

1993 Reference Link supported by the State Library begins. Consists of five regional reference 
and referral centers supplying backup reference services for public libraries. This outreach 
service was a resource for JumpStart libraries moving to the Internet. 

1994 JumpStart Advisory group started and other internal state efforts an infrastructure planning 
begun. Contract with OSU for training signed during this period. 

Fall 1995 First JumpStart grant application process announced via State Library LTLO - Letter to 
Libraries Online newsletter (see references below) and direct mail to all public libraries 
and many school libraries. For sample application form see Appendix J-l. 

1995 Prior to the first bootcamp Diane Hall, the State Library Reference Link librarian at OSU 
had offered regional workshops on OCLC’s FirstSearch, Netscape, and Internet resources — 
the Internet was not completely new to most librarians. 

12/95 Ruston Brandis, State Library Network Consultant goes to his first planning meeting at 

OSU, topic the State’s Infrastructure grants ($10,000 grant for equipment minus training and 
technical support) which preceded and then paralleled the JumpStart project 

12/95-3/96 A series of planning meetings take place picking sites, picking hardware and software, 
planning for the Internet bootcamp, and other support activities. 

12/28/95 PORTALS, using the HEA II-B grant, funded 10 of the 25 public libraries connected as part 
of the State Library’s Jumpstart I. These libraries were: Chiloquin Branch, Klamath 
County Library; Driftwood Library of Lincoln City; Dufur School /Community Library; 

Elgin Public Library; Illinois Valley Branch, Josephine County Library; Langlois Public 
Library; Lebanon Public Library; Nyssa Public Library; Port Orford Public Library; and 
Seaside Public Library. The State library handed 11 additional JumpStart libraries from 
other sources. In addition, the State Library handed 25 more JumpStart II libraries using 
LCSA funding and other sources. Eight libraries that did not require training or support 
received $10,000 infrastructure grants for equipment (funded by the State Library). 

1-2/96 Rush ton Brandis identifies Internet service providers for each library, negotiates 

individual contracts for each library, then tells each library who to call. Rushton Brandis’ 
efforts enabled the libraries to avoid a major common difficulty in the connection of the 
rural libraries to the Internet process. His background as both librarian and former vendor 
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representative was a key asset. Subsequently, the State Library (via Rushton and the 
Grants Consultant) sent the monthly connection fee to each library and the library in turn 
paid the Internet Service Provider. While awkward, libraries had leverage with ISPs and 
gained a better sense of the costs involved. In the few cases (3 of the 25 sites in JumpStart I) 
where the nearest ISP was a long distance call away, the State Library paid for the long 
distance call. The original intent was to work with the Association of Oregon Independent 
Telephone Companies as the ISPs. This did not work (except in two cases. Halfway and 
Pine) because the independent phone companies were not prepared. Rushton Brandis used 
Mecklermedia's The List (http:/ / thelist.intemet.com/) and the Oregon Online Oregon 
Internet Service Providers list (http://www.state.or.us/provider.htm) to identify ISPs. 

He found that the rates and how they were computed varied greatly: $1 per hour in 
Halfway, $30 /month for 60 hours, to $20 a month unlimited. The number of hours connected 
to the Internet varied with each library. Some libraries (e.g.. Driftwood) provide Internet 
access whenever the library is open, other libraries offer fewer hours of access. 

1/96 Charlene Grasse, project manager at OSU, convened the Internet Bootcamp Project Team to 

plan first two-day training session and hardware /software purchase and installation. The 
team consisted of two software consultants, three OSU librarians, Diane Hall a librarian 
from Reference Link (a state Reference Referral Center program), and two Research 
Assistant automation specialists. OSL sub-contracted with OSU because it had the needed 
facilities, had done Internet training, was knowledgeable about Internet resources, and the 
State Library had one of its Reference Link nodes at the OSU library. 

1/96 The Oregon Public Library Home Page for JumpStart libraries created. Available: WWW: 

http://govinfo.kerr.orst.edu/jumpstart/jump.html All participants viewed this website as 
the principal way of providing ongoing support to JumpStart libraries. The website 
contains links to web sites of interest, e-mail form for communication with other JumpStart 
participants and OSU technical support. 

3/96-12/96 Stephen Mosley hired by OSU to provide technical assistance to the JumpStart libraries. 
Introduced at the first Boot Camp. 

3/18-19/96 First two-day JumpStart bootcamp training session given at Valley Library OSU. Twenty- 
four libraries participate. Topics covered include presentations by various key participants 
including the State Librarian, State Library Network Development Consultant, and OSU 
technical support staff; introductions to Netscape, information sources on the Internet, 
library catalogs on the Internet, the Government Information Sharing Project; a session on 
dialing into the Internet service providers, hands on sessions including introduction of the 
equipment to be used at each library; and an introduction to potential policy issues, 
problems, and solutions. Each library received its equipment (and transported it back to the 
library) at the end of the bootcamp. 

4/96 Internet service to officially start at each JumpStart I library. 

7/96 Application process for JumpStart II begun. 

8-9/96 Rushton Brandis again makes arrangements for JumpStart II libraries connections with ISPs. 

In this case due to budget restrictions the State Library paid all the bills (rather than the 
individual libraries). 
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10/96 Second two-day JumpStart bootcamp training session given at Valley Library OSU. 

Twenty-two libraries participate. Second session includes participants from the first 
bootcamp to discuss their implementation experiences. 

11/96 Proposition 47 passes, a statewide "cut and cap" initiative that will severely affect library 
budgets. 

12/% Technical support contract with OSU ends, OSL picks up technical support. 

4/97 With funding for the JumpStart I libraries about to end and Proposition 47 reducing funding 

to public libraries across the state, the State Library calls each of the JumpStart libraries 
to see if another year of funding is needed. All libraries say they do not need funding they 
have made their own arrangements with local Internet Service Providers. 

4/97 Stephen Mosley re-hired at OSU, this time as a CGI programmer. 

5/97 Emporia State University (Kansas) takes over support of the Oregon Public Libraries 

JumpStart home page as principal person at OSU leaves to go to library school. 

6/97 Feedback from the two bootcamps indicates the large range of skill levels among the 

participants; all liked the chance to interact with fellow librarians; wished there were 
better ways of anticipating use of the service; there was high interest in security and 
filtering software (in the case of filtering, some did, most did not); there was keen interest 
in sources for dialing in and reading e-mail; and ISPs ranged in the quality of phone lines, 
service, software, and costs. 

6/97 The Oregon Regional List of Serials (ORULS) becomes available: WWW: 

http://www.osl.state.or.us/oslhome/orulshome.html WWW: 

http://db.osl.state.or.us/orulsl.htm A major statewide library resource providing an index 
to periodical and other serial holdings of 160 Oregon (and Washington) libraries. ORULS- 
Web contains approximately 250,000 holdings of 100,000 titles converted from MARC tapes 
and mounted on Windows NT server, using Inmagic software. Installed on machine 
purchased as part of the grant. Used regularly by JumpStart libraries and others. 

Seaside Public Library’s Experience 

1995 Reita Fackerell, after 8 years working in the library, becomes library director. She had 

her own e-mail account through EdNet prior to the grant. 

Fall 1995 Reita Fackerell submits a JumpStart grant application, but would not have if the 
application was more complicated. For sample application form see Appendix J-l. 

12/95 Library Director notified that Seaside library had received an award. 

l/% Library Director and staff plan for staff responsibilities and location of the workstation. 

2/96 Library Director develops acceptable use policy approved by the library board on 3/1/96. 

Most useful was to download examples of policies by other libraries from the Internet. 

Board excited and supportive. 
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3/18-19/96 Internet Bootcamp attended by Reita Fackerell and Paula Clark local JumpStart Project 
Manager 

Library Director only waited a couple of days after returning from bootcamp to begin public 
access so that she could train her staff but then went public. 

Began publicity in local papers. 

The Library Director made a presentation to the Seaside City Council, their chief concern 
was stopping "minors access to dirty pictures," but in general all excited. 

Library Director and Project Manager begin staff and volunteer training in emest. Reita 
commented that even now about half love the Internet and half hate it (and she does not 
push it). 

Library Director estimates that it took 6 months for the service to take off. Initially, and 
unexpectedly, seniors (requesting health information) were primary clientele (75% of 
initial users). In the recent six months kids are heavy users (games, chat sessions, and 
homework help). 

Driftwood Public Library, Lincoln City Experience 

For Background on the Driftwood library see the Driftwood Library Gazetteer Special Edition, 
Appendix J-5 

1965 Driftwood Public library begins. 

1967 Library becomes a department within city government. 

1980 Sue Jenkins is hired. 

1993 Library moves to its present facility which tripled its space and raced circulation by 40%. 
12/93 Yueh-lin Chen is hired as cataloger. 

1994 JumpStart Advisory group starts, Sue Jenkins is a member representing small libraries. 

1994 Sue Jenkins starts using the Internet, America Online is the only provider available. Her 

network use is in conjunction with obtaining her MLS via an Emporia State University 
distance education program. 

Fall 1995 Sue Jenkins applies for JumpStart grant. 

12/95 JumpStart I grant awarded to the Driftwood library. 

3/18-19/96 Sue Jenkins and Yueh-lin attend Internet Boot Camp. Both find it very useful. 



3/23/96 

3/96 

3/96 

4/94 

9/96 
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3-9/96 The library kept the workstation in a back room while developing training materials, 
training staff and volunteers, and seeking board approval of policies. The Internet 
represents a large technological advance for the library: there is no computer at the 
reference desk nor cd-rom databases present. Sue Jenkins described the library board as very 
conservative and cautious. For example, neither the staff or public have e-mail access 
(board is concerned with e-mail liability), use of the Internet for entertainment, games, or 
chat is prohibited, children under 13 need parental permission to use Internet (corresponds 
with local school policy). In addition, all new Internet users must take a one hour lesson 
before use. Driftwood did not use four months of the connection paid for by the State 
Library fully because there was no public access permitted. Driftwood kept the equipment 
in the technical services area during this period, trained staff, and sought approval of 
public access policies. Sue Jenkins recognizes this approach is different from other libraries 
but believes the approach justified given the community (and that the introduction of the 
Internet was successful). 

3-9/96 Volunteers recruited and trained. Ten volunteers trained, 6-7 are regulars allowing for 2 
people covering 4 hours each day the library is open. Volunteers train users, are subject 
experts, and are given assignments to check out new sources on the Internet. 

9/96 Workstation moved to its present public location in the reference area to "mild fanfare." 

Sue Jenkins, exhausted by the preparations necessary to get the workstation to the public, 
did not spend as much time as she had planned advertising the services availability. 

Over half of initial use was by seniors. Other users included local small business (do I want 
the Internet for my office), genealogists, and the curious. Few students use due to restrictive 
policies. At present 60% are seniors, 30% teenagers (an alternative high school is in the 
same building), and 30% are middle aged (small business, stock quotes, tribal interests). At 
any point in time 80% of the users are new, 20% repeat users. 

9/96 Sue Jenkins designed use policies using Corvalis and Salem Public Libraries policies. Both 

Seaside and Driftwood libraries use a signup sheet (user one day in advance signs up for one 
hour time slot). At Driftwood, users must take a one-time, one hour lesson conducted by 
librarian or volunteer in advance of use. 

6/97 Of the 8 staff in the library, 2-3 have e-mail accounts at home at present. 

Staff Development Activities 

This section summarizes efforts by grant participants to obtain education and training for staff 
to better implement the grant objectives. 

Reita Fackerell’s, Seaside Library Director, chief source of continuing education is the Internet. 
In particular she made use of the Internet to download library policies in a wide range of areas and used 
them to create or update her own library's policies. Computer manuals are second in popularity! Also 
helpful are a range of listservs including stumpers-1, publib, Oregon library lists, etc. The bootcamp was 
great. The JumpStart home page is heavily used by Reita and users (search engine section the most 
popular, followed by Federal statistics like CPI). Reita did not find information in magazines and 
journals either accessible or helpful. There is a local community college which offers an introduction to 
navigating the net and an introduction to HTML (which she plans to take). 
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Sue Jenkins completed two masters degrees (MLS from Emporia State and an MPA from Lewis 
and Clark) during the grant period. The Emporia program forced her to get involved with the Internet 
early (the program requires all students to have e-mail accounts). Courses and contacts with instructors 
gave Sue a better than average knowledge of the literature and latest professional thinking. Sue found 
web pages and listservs (digi-lib, adapt-1, state library lists, and Emporia State alumni list) to be most 
helpful. She also found the ISP to be a good source of information. Sue also participated in various 
state workshops and meetings (on the JumpStart advisory committee, reference roundtable, attender the 
state library association workshop on the web). Sue strongly believes in staff training and is lobbying 
locally for the creation of a city-wide mini training lab where city staff would regularly update their 
information technology conceptual and skill base. Yueh-lin Chen also found a network of fellow Queens 
College graduates communicating by e-mail to be of particular help. 

Project Work Products 

This sub-section lists and describes the major products, services, and other outcomes as a result of 
Jthe grant: 

Oregon Public Libraries (Jumpstart) homepage. Available: WWW: 
http://govinfo.kerr.orst.edu/jumpstart/jump.html 

PORTALS HEA II-B grant provides seed money to fund the connection of 10 rural public libraries to the 
Internet including: Chiloquin Branch, Klamath County Library; Driftwood Library of Lincoln City; 
Dufur School /Community Library; Elgin Public Library; Illinois Valley Branch, Josephine County 
Library; Langlois Public Library; Lebanon Public Library; Nyssa Public Library; Port Orford Public 
Library; and Seaside Public Library. For a complete list of all libraries connected via the Oregon 
Connectivity grant project (of which JumpStart was part) see: Oregon Internet Connectivity Grant 
Project Sites, Available: WWW: http://govinfo.kerr.orst.edu/jumpstart/ormap.html the State 
Library using other sources funds the other JumpStart I participants and all of the JumpStart II 
participants. 

PORTALS, using the HEA II-B grant to fund 10 rural JumpStart libraries, served as a catalyst to 
connecting the rest of Oregon’s public libraries. To date 46 rural public and school libraries have 
Internet connections. In addition, eight libraries that did not require training or support received 
$10,000 infrastructure grants for equipment (funded by the State Library). Altogether 95% of Oregon's 
public libraries have Internet connections. 

Dissemination of Project Results 

This sub-section identifies specific ways grant participants disseminated project findings to the 
local, state, and national communities and the profession including publications, videos, press releases, 
presentations, external training sessions, etc. 

State Library 

The State Library made active use of its library consultants, its LTLO - Letter to Libraries 
Online newsletter, Oregon Library Association meetings, and direct mailings to communicate with 
JumpStart libraries. Later the State Library used data reported by JumpStart libraries with database 
vendors to indicate that they did not really know what public libraries needed or wanted. The State 
Library did not have any contact with other places doing similar projects. 
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osu 

Stephen Mosley from OSU visited the JumpStart libraries for technical support. OSU 
communicated with JumpStart libraries via the Oregon Public Libraries web page (used heavily by 
both libraries interviewed), listserv (not used at all by the libraries visited), e-mail, and telephone. 
OSU produced the only article for national dissemination of project results so far: 

Middleton, Cheryl and Cross, Judy. (Forthcoming). Connecting rural public libraries to the 
Internet; or "Will it fit in my car?" Public Libraries, (manuscript version included as Appendix 
J-4) 

A group affiliated with the University of Southern Louisiana recently visited OSU to find out how the 
JumpStart program worked. 

JumpStart Libraries 

Seaside and Driftwood Libraries made active use of local media including newspapers, radio 
and cable TV interviews, brochures (for an example see Appendix J-6), and photographs. The State 
Library required JumpStart libraries to include press release materials in their monthly evaluation 
reports. The State Library tracked mentions of the JumpStart program in Oregon newspapers. 

Evaluation Efforts 

This section identifies specific ways each participant used to evaluate the project. The State 
Library filed the report required by PORTALS and the grant Evaluation Consultant in February (Perez, 
1997) noting on p. 3 that 

The project supervisor is tracking progress of the Jumpstart grant recipients. Program 
participants are tracking and making monthly reports on evaluation indicators, including 1) 
number of public access sessions, 2) number of connect hours, 3) number of library program 
presentations relating to the service, 4) publicity releases to local media, and 5) "success story" 
anecdotal reports. 

In addition, OSL regularly contacted the JumpStart libraries via e-mail, phone, and fax. In April 1997, 
OSL called all JumpStart sites to inquiry about the program's success and to find out if there was a need 
for additional funding due to Proposition 47 cutbacks. OSU conducted it own survey evaluation of the 
JumpStart libraries after each bootcamp regarding its effectiveness (for a sample survey response see 
Appendix J-7). The State Library Reference Link regional sites noticed an increase in use particularly e- 
mail from JumpStart libraries attributable to JumpStart connections. 

The local libraries interviewed gathered much of their evaluative information as part of the 
face-to-face interaction with users during training sessions. In one case, the local library collected the 
evaluative data required by the state but did not use the data itself. In the other case, the library did 
not collect or report the evaluative data. There appeared to be no systematic attempts to use this data 
to step back and assess the impacts of the service on the library, librarians, or users. 
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Lessons Learned 

This sub-section identifies significant lessons learned by the grant participants which may also 
be of interest to similar organizations in other settings. 

Simplified Grant Application Form a Big Plus 

The JumpStart grant was the first grant Reita Fackerell, at the time a newly appointed Library 
Director of the Seaside Public Library, applied for as a library director. She viewed any grant 
application process as daunting. Yet, when she received the grant application, she found the 
application was straight forward, "just fill-in-the-blank...a no brainer." This simplified approach 
was a big asset given the targeted group for the project. 

Computer Literacy a Must 

The libraries that had the most problems were the ones in which those involved had the least 
(or no) computer literacy skills. JumpStart organizers thought they had solved the problem by 
permitting anyone from the community to be project manager not just librarians. But in some communities 
either no computer literate people existed or the library found no one to participate. In these cases, 
intensive literacy training would need to be done first prior to the Internet bootcamp. In the future, 
further effort would be made via phone or mail survey to identify the computer skill level of bootcamp 
participants in advance. 

Internet Bootcamp 

The bootcamps were very successful. OHS and the State Library managed a range of potential 
issues including: coping with the diverse range of computer/Intemet knowledge among bootcamp 
participants, distance to the bootcamp training site (a problem for some but all made it), inability to do 
hands-on training of dialing up Internet Service Provider (as they were reluctant to release passwords), 
inability to connect to the Internet immediately upon returning from the bootcamp (due to wiring, phone 
line, and other technical problems — eventually resolved). The decision to give the bootcamp 
participants their equipment at the end of the weekend was a useful motivator to attend and learn. 

Bootcamp participants, in hindsight, wished for more exchange with librarians experienced in 
introducing the Internet into their communities; role-play at the keyboard of how a librarian might 
introduce the Internet to a new user (in particular Reita stressed the need to learn how not to make 
anxious new users feel like they are dummies); more detailed instruction on Internet tools: ftp, telnet, 
gopher, and search engines (Middleton & Cross, forthcoming); more Internet "surfing" experience (using 
the Internet as a reference tool) with experts present to troubleshoot and guide; both libraries 
encountered new and unexpected user information needs, prominent among them was the need for health 
information; and better solutions to their security concerns (discussed below). Several felt more 
attention should be given to hands on troubleshooting of equipment including nitty gritty like 
diagnosing common problems, how to install RAM and a hard disk, etc. Sue Jenkins stressed the need for 
good search skills (beginning with online searching and moving to Internet searching). Rushton Brandis 
would insist on basic computer literacy skills in advance of the bootcamp. Those who did not have the 
basic computer literacy skills would be required to attend a computer literacy workshop prior to 
participating in the JumpStart bootcamp. 
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Turnkey Approach 

The decision to provide everyone with the same pre-tested equipment with the same pre- 
installed software paid repeated dividends beginning at the bootcamp (all could be trained on the same 
hardware and software) and later when providing technical support. Volume purchasing reduced 
hardware and software costs. The choice and pre-configuration of software allowed the local libraries 
to, in general, simply turn their equipment on and be automatically connected to the Internet. 

Rushton Brandis (among others) debated the buy local v. turnkey approach. The need to 
standardize for technical support and reduced cost due to volume buying swayed him to the turn key 
approach. He did recognize that this approach did not promote local library - dealer ties and that 
local dealers would be more reluctant to repair defective equipment (which was an issue in Seaside and 
Lincoln City). 

All in all Ruston Brandis and the OSU staff are to be applauded for their careful choices of 
hardware and software within a tight budget. Involving experienced OSU purchasing agents made an 
important difference. 

Buy the Best Technology and Connection 

Sue Jenkins, in particular, mentioned that she felt the technology was under powered. Sue’s 
library was in the first JumpStart cohort (their equipment is now more than a year old) and some of the 
problems she mentioned the State Library (or OHS) fixed in the second cohort's equipment. Sue is in 
the process of upgrading her equipment including replacing her hard disk (nothing but problems), 
increasing RAM to 32 MB, adding a graphics printer (she has one user who needs to download star maps 
and couldn't, and other who wishes to download graphics), would like to get a faster processor (was 
upgraded to 133 MHz for JumpStart II libraries), and a larger 17" (was upgraded to 17' for JumpStart II 
libraries) or 20" monitor (the 14” monitor is too small for the seniors who are prime users and for group 
training sessions). None of the above comments were meant as criticism of OSU and the State library 
who as both noted "did the best they could with the budget they had." The evaluators make these 
observations in the context of advice to others planning a similar project elsewhere. 

Both libraries thought it was a good idea to have the Internet connection active every hour the 
library was open. Several librarians (and Internet service providers) wondered whether an ISDN 
connection would be better (all complained of slow system response, the onsite evaluator also noted slow 
response as well). Rushton Brandis noted that OSL did not employ ISDN connections due to the lack of 
general availability and high cost. Some vendors proposed the use of dedicated 28. 8K lines in a few 
cases. 



The State Library would consider paying for a second phone line for the library as part of the 
connection package (libraries would later pick up the cost) for future grant programs. For many libraries 
the second phone line represented a significant initial barrier. But the State Library felt for JumpStart 
I and II that the additional phone line should be part of the libraries contribution to the project. 

Technical Support a Central Requirement for a Successful Program 

First, the recognition of the need for initial, ongoing technical support should be commended. 
Second, OSU's Stephen Mosley brought at least three key assets to the technical support he provided, 
as reported by the librarians the evaluator visited (and echoed by the State Library Networking 




40 



Ryan /McClure Onsite Evaluation Report: Citizen Access to Government and Other Information August 12, 1997 



Consultant): 

• He was technically knowledgeable. 

• "He never made me (the local librarian) like an imbecile no matter what I asked!" 

• He was genuinely committed to the success of each library in resolving its technical 
glitches. 

Stephen Mosley in his technical support role supplied a key ingredient in the success of the JumpStart 
project despite being in a position with many incentives to ignore a request for technical assistance or 
merely respond in an unhelpful, safe, bureaucratic, fashion to often poorly expressed technical needs 
from remote locations. 

The OSU technical support contract ran roughly from March - December 1996, nine months for 
the JumpStart I libraries and three months for the JumpStart II libraries. The State Library assumed 
the technical consultation role at the end of the OSU contract but at a reduced level. The State Library 
Network Development Consultant reports few requests from the JumpStart I libraries and diminishing 
requests from JumpStart II libraries. Three JumpStart II sites have persistent problems to be resolved. 
The need for, and length of, technical support appears to vary greatly from library to library. All 
benefitted from an initial period of technical support including the capacity for onsite visits as well as 
telephone and e-mail consultation. The ending of the technical support provided by OSU, however 
valuable, forced the libraries to make local arrangements for technical support or otherwise become 
self-sufficient. This development is a necessary transitional step. The evaluator would tentatively 
suggest, based on the JumpStart experience, that technical support for six-months is adequate in most 
cases. 



The evaluator explored with the librarians at the two libraries visited the feasibility of a 
group of Oregon libraries cooperatively arranging for technical support. The Seaside librarian 
promptly responded that she would willing commit $300 per year for technical support like that 
provided by Stephen Mosley (one site visit per year — additional visits on a pay as you go basis and 
unlimited telephone and e-mail consultation). The Driftwood (Lincoln City) librarian said that they 
had made local arrangements and would likely not participate. As suggested below, cooperative 
technical support bears further investigation. 

An important early decision was to standardize the hardware and software used. This decision 
made it possible to provide meaningful technical support (probably the only way to do it). However, 
this made it difficult for libraries who used different hardware (Mac) and followed different software 
upgrade paths (Sue Jenkins moved from Windows 3.1 to Windows 95 which worked noticeably better but 
which Stephen Mosley did not know). 

Time and Timing 

Even the most successful libraries took months after the bootcamp and receipt of their 
equipment to either begin public access to the Internet (many wished to train their library staff before 
going public, others faced wiring or phone line problems, still others had to negotiate various library 
board approvals) or get the type of public use of the Internet service they expected. For most public 
librarians, attention to any one area (like the introduction of a new Internet service) is, at best, 
episodic. Most librarians did not credit the time necessary to simply reduce their own, their staffs, and 
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their board's (and other key stakeholder's) ignorance. Sue Jenkins remarked that libraries should 
expect more use than they imagined (involving more staff time than expected), but that use takes time 
to build. 

Yueh-lin Chen remarked that her library spent months worrying and trying to prepare for 
every eventuality only to discover that none of the problems anticipated occurred. She along with 
Reita Fackerell at Seaside Public Library advocated waiting for a problem to occur before developing 
policies and procedures rather than doing too much pre-planning. 

Policy Issues are Local and Take Time to Work Out 

The principal problem faced by many libraries was dealing with the question of minors access 
to pornography. Developing policies took more time than expected. Different locations developed 
different policies depending on local circumstance. In some cases, the State Library asked the State 
Library's Intellectual Freedom Committee to work with local communities. Librarians who had first 
hand experience with these and other issues discussed them at the Internet Boot Camp. In sum, be 
prepared for a time consuming but necessary policy process, allow for local solutions, prepare librarians 
for likely issues and current professional thinking and resources, use local professionals experienced 
with the issues, and provide backup support where needed. 

Position the Workstation Directly in the Public Eye 

Prior to the introduction of the service in the JumpStart libraries there was great worry about 
inappropriate use of the service (i.e., exposing minors to pornography). In the two libraries visited, 
these concerns proved to be non-issues. The librarians attribute the appropriate use of the service to two 
decisions. First, placement of the Internet workstation in a prominent public location. One librarian 
remarked. "If you are going to be using pornography, the whole world will know that you are." Another 
librarian noted that public placement, made librarian supervision of the workstation easier. A second 
factor was the extensive prior discussion among library staff and library board (and other key 
stakeholders like city government officials) about what would constitute acceptable use. This policy 
making process contributed to the establishment of shared norms of behavior and appropriate sanctions 
while educating key stakeholders to the issues. Prominent placement of the workstation is a great 
advertisement for its use. 

Internet Access: Matching Reality to your Own, your Staff, and your Users Expectations 

A JumpStart library is seen as the central place in the community to test drive this "new- 
fangled Internet thing" with several important consequences for libraries: 

• JumpStart librarians must pro-actively take the lead in establishing norms and 

expectations for this new information aid. Most users come to "test-drive" the Internet at 
the library with inflated expectations of what the Internet can do for them. Both sites 
visited, independently mentioned the Intel TV commercials as a principal inflationary 
culprit. When users use the Internet at the library their expectations are not likely to be 
met. Users will blame themselves ("I’m just dumb when it comes to computers.") or the 
library (the technology is not state-of-the-art), or the librarian (the librarians are not all- 
knowing). Solutions to this situation are more difficult due to: librarians inevitable 
feelings of inadequacy, less than state-of-the-art equipment, public pre-conceptions about 
libraries, etc. Successful librarians recognize their leadership role in Internet use norm- 
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setting. Successful librarians move their community's focus from a norm based on a present 
oriented imperfect product (the library's workstation, librarians uncertain knowledge, and 
the user’s anxiety) to a perfectible learning process (where it is ok to be imperfect if there is 
tangible commitment by the user, librarian, and library to improvement). 

• JumpStart libraries must seek fair compensation for its Internet sales and training role. 
Libraries introduce community members to computers and the Internet, they train the 
community in Internet use, and serve as a resource for technical problem solution and 
information resource finding. The library's purchase of equipment or services from certain 
companies is perceived as a strong endorsement of that company. In essence, the library is 
serving as an unpaid member of the Internet service providers’ and computer stores' sales 
and technical support departments. Compensation should be sought not only from the 
businesses involved but also from the community which must recognize the need for a 
technologically literate workforce. Public spirited ISP and computer stores often give 
libraries discounted rates. But are these companies aware that discounts to libraries make 
good business sense? At minimum, libraries should obtain free connection to the Internet 
during their public hours of operation. From an ISP perspective such an investment pays for 
itself quickly. 

• Libraries must recognize that the training of new Internet users will not go away. Both 
libraries visited noted that there were fewer than expected regular users of the Internet 
service (estimated by both libraries visited as 20% of total). Instead there was a constant 
stream into the library of new or novice Internet users. Most users who found the Internet 
useful purchased their own (office or home) equipment and services to access the Internet. 
The costs of continuously training new users is high in staff time, training materials, and 
inefficient use by novice users of technological resources. The point is not that libraries 
should not be providing this service. The libraries visited would not consider ending their 
Internet service. The point is that libraries, particularly those offering the service for the 
first time, should know who their likely clientele will be and the resulting likely cost 
areas and costs. 

JumpStart libraries were slow to recognize and plan for these costs. 

E-mail Access 

The decision to allow staff or users to have e-mail access remains unresolved at many libraries. 
The libraries’ concern is with inappropriate use by patrons and the resulting liability of the library. 

The issue is somewhat resolved by telling users of various services offering free e-mail (for a trial 
period). Reluctance by these libraries to get involved with e-mail encourages interested users to get 
their own accounts with the local ISPs. 

Hidden Costs 

There were several unanticipated costs to libraries when connecting to the Internet which were 
uncovered including: costs to install additional wiring for power to the workstation, cost of an 
additional phone line, cost of reproducing brochures and Internet training materials (Sue Jenkins printed 
over 400 copies of a Internet workbooks - money well spent, but expensive if not in the budget), printer 
toner and paper costs, unanticipated equipment repair and software update, staff time required to be 
introduced to the Internet, staff time needed to train users often one-on-one, and the high cost of 
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obtaining help to repair potential broken computer components (charges of $75-100$ per hour seem the 
norm). Reita Fackerell’s annual costs include $40/month for extra phone line, $250 for Internet access 
from ISP, $99 for toner cartridge 4 times a year, $200 to unexpectedly replace a broken printer drum, and 
$100 for an unexpected modem repair. Both Sue and Reita suggest budgeting for unanticipated repairs 
and equipment upgrades. Both believe a backup workstation (also used for staff) would be useful. 

Role of Volunteers 

The JumpStart libraries visited appear to have had little problem recruiting a local cadre of 
volunteer Internet trainers. Sue Jenkins at the Driftwood Library developed a training package which 
she uses with staff, volunteer trainers, and library users (see Appendix J-8). She found enough 
volunteers to offer user training for several hours each day. Sue Jenkins or Yueh-lin Chen train 
volunteers and then schedule them to train users during specific hours each week. In addition. Sue 
Jenkins gives volunteers assignments to check out new Internet resources and assigns each volunteer to be 
a subject expert in specific areas. While some volunteers are simply heavy users of the service, most 
play a critical role in training fellow community members in Internet use. 

User Training: An Important, Learned Skill 

Training of Internet end users apparently did not receive enough attention at the bootcamp. 

Most JumpStart libraries learned how on their own by first training themselves, their staff, their 
volunteers and then their users. This process while laborious and time consuming did increase staff 
ownership and helped to embed the service within the organization. The staff, together, figured out 
what the Internet was and how to tell others what mattered about it. Sue Jenkins produced a training 
manual for use in required one hour user training sessions (see Appendix J-8). The manual was as much 
use in training library staff as it was to users. Yueh-lin Chen remarked that her library learned the 
hard way to keep handouts short --otherwise patrons will not read it. Reita Fackerell stressed 
learning the art of "never seeming to be bothered" when users asked for help. New users were sensitive 
to asking for too much assistance from anyone. Reita also stressed the importance of learning the non- 
verbal cues (screen frozen for 3 minutes) suggesting a patron needs help. All wished for more instruction 
in this area. 

Ongoing Security Concerns 

Both libraries visited by the evaluator identified problems with users tampering with various 
customizable features of the application and operating system software. Uniformly these "security 
breaches" were not done by malicious "crackers." Some users sought to customize the system to the 
settings they use at home, some were impatient with the slow system response time, other users were 
merely curious. Both libraries were from the first JumpStart group. The second JumpStart group 
received a security software fix. It is unclear whether the security software meets the libraries needs. 
The State Library Network consultant did report that several of the small rural libraries that received 
the security software found it confusing and had it disabled. The two phase one JumpStart libraries the 
evaluator visited appear not to know about the software fix (perhaps in part because the JumpStart II 
libraries' experience with the security software was so mixed). 

Unexpected Bonus: New Users, New Awareness of Community’s Information Needs 

Both libraries independently remarked that this service brought many new users to the library and 
that some of these users have since used other library services. In addition, the libraries note an 
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expansion of their awareness of their community and its information needs. For example, neither 
library realized the extent of its retired community, nor their interest in health information. The 
introduction of the Internet brought these users and their information needs to the fore. 

Reasonable Rules, Reasonably Enforced 

The JumpStart project rules seem both thoughtfully constructed, minimal, and reasonable 
beginning with the simplified application form, continuing with requirements to prepare the site 
(wiring and phone), attend the bootcamp, prepare a locally endorsed, acceptable use policy, conduct 
ongoing publicity campaigns, and file monthly evaluation reports. Enforcement, when the letter of the 
rules was not met by a participating library, was most often in the form of additional help and 
guidance. 

Evaluation the Key 

The State Library and OSU devised a useful mechanism to begin to pragmatically evaluate the 
JumpStart project. The participating libraries collected a wealth of data on their experiences bringing 
the Internet to their communities. But has anyone systematically analyzed their evaluative data to 
discover the story it tells? Now, at the successful conclusion of the initial flurry of activity, is the time 
to examine this data. OSL and OSU may have the skill to analyze the data, but what of the 
participating rural public libraries? The Onsite Evaluators initial impression is that the JumpStart 
libraries would now benefit from an evaluation workshop. The objectives of the workshop would be to 
explain in pragmatic terms what evaluation is and how it is done, convince the participants that 
evaluation pays dividends, begin to evaluate some of the local data already collected, and collectively 
devise efficient and effective evaluation mechanisms that will work upon their return to the local 
setting. Why collect the data and then not analyze it or train local librarians to interpret the data 
collected? 

Successful Partnerships Combining Enlightened Self-Interest and Service 

The PORTALS use of HEA II-B funding to the State Library for JumpStart provided crucial, 
timely, seed money to get the project started. The State Library with its ties to the Oregon library 
community successfully assumed a leadership role coordinating and smoothing the diverse needs of the 
participants and committing its own resources when the project would otherwise founder. Notable was 
the Network Service Consultant's negotiations for provision of Internet connection service on behalf of 
each JumpStart library with the numerous Internet Service Providers. The project could well have 
foundered or come to a halt without these negotiations. 

The State Library's partnership with OSU was another key element in the project’s success. 

The State Library recognized that OSU’s Valley Library staff possessed the additional purchasing, 
technical support, and training expertise needed to make the JumpStart project succeed. OSU was 
already recognized as a trusted provider of reference information through its participation in the State 
Library’s Valley Link Reference and Referral Center program. OSU is a land grant university with an 
incentive to serve a wider audience than simply its academic community. But the OSU library also 
recognized that in order to justify the resource commitment needed to fund a move to a digital library 
environment there must be more than a Internet connected university - the state s citizens must be 
connected as well (see Middleton & Cross, forthcoming). 
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The connection of JumpStart libraries provides leverage to the State Library in the database 
licensing negotiations with various database vendors. The State Library can point out the 95% rate of 
Internet connection among the state's public libraries and that their needs for information (as 
documented in their JumpStart reports) are far more diverse than previously thought. 

In some cases, the JumpStart library's experience prompted other institutions, like city 
government in Junction City, to connect to the Internet as well. The paid JumpStart library connection in 
some rural locations prompted ISPs to venture into new geographic areas that would otherwise be 
ignored. 

Next Steps 

This sub-section suggests next steps to be taken as this grant period comes to an end and to 
advance the project's objectives after the funding period. 

Local Libraries 

Both of the libraries visited by the evaluator independently identified similar next steps: 

• Libraries as servers and publishers not only clients and intermediaries. The sites visited 
both had active plans for a library homepage. Priorities were to make the library catalog, 
databases, hours, and programs available conducting interactive or online reference and 
referral assistance, and with forms for book renewals, signups for library programs and 
other library services, and interlibrary loans not far behind. Libraries might also consider 
being sites for provision of city government information and services as well. There appears 
to be a window of opportunity for rural libraries to establish themselves in these rural 
communities as the local government information technology experts. This moment should 
be seized if possible. Librarians will need training in this effort, an area the evaluator 
discussed with the OSU library staff responsible for the bootcamp. Librarians will also 
need space on a shared server, at least initially. Local Internet service providers offer one 
obvious source. Another might be a shared State Library site - with present technology, 
the server need not be local. One unexpected area in which some librarians will need 
training is in the identification of local information to include on the library server. These 
are largely mindset problems: some librarians are too book oriented, others are not trained 
in the creation of information only in its organization and verbal and written presentation. 

• Develop Local Area Networks (LAN). The two libraries visited were considering LAN 
development in partnership with city government (both were officially units of city 
government). Both recognized the benefits of shared resources via LAN connection. The 
evaluator pointed out potential benefits of local links to other non-profit organizations as 
well (e.g., local visitor and convention board, chamber of commerce, etc.) 

• Upgrade Internet connection to 56K and eventually T1 leased lines. The existing connections 
are slow, users require higher bandwidth (with graphics and eventually sound), and the 
prices on faster service is dropping. The two libraries visited are ripe for the next level of 
improvement of their connection. It may well be that more lines, in addition to upgraded 
lines may eventually be needed as well! 
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• Purchase of an additional Internet workstation. This work station would serve as a backup, 
be used by staff, and be available for increased user demand. The public has more access to 
the Internet than the library staff at some locations. In one case, the Library Board was 
going slow on allowing staff to have e-mail accounts -- the fear being inappropriate use. 

All of these steps are clear indicators of successful adoption of the Internet service by these JumpStart 
libraries. Both libraries face a tight budgetary reality, particularly over the next fiscal year. Sue 
Jenkins strongly advocated for the library as being the information technology training center for city 
government (and the community). She is seeking funds to create a mini training lab. 

State Library 

The State library is presently planning a third JumpStart phase connecting more school 
libraries and rural schools to the Internet. A principal future focus now that libraries have Internet 
connections will be increasing the useful content available. Recently the State Library made the 
Oregon Regional List of Serials (ORULS) available on its web site (this is housed on a computer 
purchased with grant funds). This is a major aid to resource sharing and interlibrary loan and will be 
regularly used by JumpStart libraries. The State Library as part of a State-wide database licensing 
committee is completing negotiations with UMI on-providing web based access to the full text of the 
Oregonian (Oregon’s principal paper) and several other IAC indexes to popular materials. There is 
also interest in adding library catalogs to the State Library web site with various possible schemes 
being actively pursued. The library is looking closely at the impact of the Telecommunications Act 
(P.L. 104-104) discounted rates (may mean 70-80% discount for rural Oregon libraries). There is the 
expectation that within two years JumpStart libraries will move to Tl lines (some already have). It is 
unclear who will fund the migration to the faster connections, federal LSTA funding is possible. 

Recommendations 

The evaluators offer the following recommendations based on the onsite visit. 

State Library 

The JumpStart project is a clear success. Efforts to continue the project's success might include: 

• JumpStart Sharing/Planning Sessions: Several librarians interviewed commented that 
they missed opportunities to meet with fellow JumpStart participants to share experiences 
and solutions to mutual problems. One or more sessions which sought to share and 
summarize the JumpStart experience for Oregon libraries would benefit everyone. In 
addition to participating libraries, Internet service providers, computer and database 
vendors, OSU, PORTALS, funding agencies and others might also be invited and play a 
structured role. One component of such a session should be addressed to assessing common 
needs for next steps as an aid to State Library planning. 

• JumpStart Server Project feasibility: Moving from client to server on the Internet is a 
natural progression for the JumpStart libraries. While there is much to consider, the State 
Library is the logical first choice to take the lead. One approach that deserves 
consideration is a turnkey approach similar to that offered in the original JumpStart 
program. The State Library could offer server space to each library with pre-scripted 
HTML template home pages to start. This approach would motivate when providing 
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training in HTML coding and introducing equipment, hardware, and software considerations 
when locally mounting a server. A number of other cooperative projects among 
participating libraries might be coordinated at the same time. 

• LAN Training: Even small libraries, particularly those who are part of a municipal 
government, now see the benefits of local area networks. An introduction to LANS for 
library managers session designed to accomplish no more than enable librarians to be smart 
users of LAN technology applied to libraries might be a good first start. Oregon libraries 
such as the Corvallis Public Library, which has a well developed LAN, might have much 
to contribute here. The State Library might broker a group discount for more elaborate LAN 
training if there was sufficient interest. 

• Train librarians how to do information technology planning: In order for a library to receive 
FCC universal fund money a library must submit its information technology plans and costs. 
Already libraries worry about what the new legislation means, how to do it etc. This is a 
perfect opportunity for the State Library to develop a program, maybe a boot camp(!), to 
help libraries to think about and develop an overall information technology plan for 
immediate and future needs while they obtain their universal fund money. If equipment 
was the carrot for JumpStart the FCC universal fund money will work here. 

• Investigate the desirability and feasibility of cooperative technical support: Is there a 
need for technical support at these JumpStart libraries? Are the libraries willing to 
cooperatively pay for it? Could the State Library broker an arrangement? The evaluators 
recommend the State Library find out. 

• Train the local trainers, train the local technical support personnel: Rushton Brandis, 
among others, recognizes that as a result of JumpStart there is a cadre of neophyte technical 
experts and Internet trainers who could be offered additional training to continuously 
enhance their skills. There appears to be a window of opportunity for rural libraries to 
establish themselves in these rural communities as the local government information 
technology experts. This moment should be seized if possible. In some cases, library staffs 
are not the local experts and need to be trained in Internet use. In either case, it would be a 
shame having identified these local experts and trainers, and provided some initial 
training, to let their skills and interest languish. Could the Internet could be used to 
deliver mini-courses and other training directly to these local libraries? Perhaps the real 
issue is who will take on the task (and fund it)? 

• Find Out What Happened, Get the Story Out! The State Library required that each 
library file monthly reports. OSU surveyed JumpStart libraries to find out about the 
suitability of training. Yet none of this collected data appears to be analyzed or otherwise 
used. Indeed, with the exception of Middleton & Cross (Forthcoming), the JumpStart story 
is yet to be told. There is much that others, in other states and around the world, could 
leam from the JumpStart experience. Get the word out! 

The State Library has taken the plunge becoming a strong force for the introduction and use of 
information technology in the rural public libraries of the state. The range of potentially useful 
information technologies is large and increasing. If the State Library does not continue to exert its 
leadership in this area, JumpStart would be an opportunity successfully created by the State Library 
and then lost. 
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PORTALS 

A new interim Executive is in place and PORTALS engaged in the process of strategically 
planning its future direction, see for example it draft strategic plan (Available: WWW: 
http://www.portals.org/plantoplan.html). As PORTALS re-thinks its role and focuses on the 
scholarly information needs of its members a project in support of Oregon's rural public libraries may be 
very peripheral to PORTALS' current interests. The notion of service to the larger community may not 
currently be a part of every PORTALS members' mission (the way it is at land-grant OSU). The need to 
create the capacity to electronically interact with communities, large and small, across state for 
marketing, admissions, and distance education probably appears too visionary for some. Certainly the 
idea that a networked environment, in which all connect in ways previously unimaginable, creates new 
unexplored opportunities and responsibilities is only now being grasped by the leading academic 
institutions in the U.S. 

PORTALS provided key funding to the JumpStart program at the critical starting point. 
PORTALS can claim its share of credit for this effort and may wish to do so more aggressively. Future 
funding possibilities for rural and public and school libraries and the Internet which require multi-type 
library or consortia participation are likely. PORTALS, building on its successes here, can play an 
active, responsible role in this area should it choose to do so. 
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IV. Multnomah County Library 

This section of the report examines the Multnomah County Library’s programmatic component 
of the HEA II-B grant including: program objectives using language from the original grant proposal 
and outcomes to date in brief, the evaluator's activities in order to compile this section of the report, a 
chronological summary of key events and accomplishments, staff development activities, project work 
products, efforts to disseminate project results, Multnomah County Library's evaluation efforts, lessons 
learned, next steps, and the evaluators' recommendations. 

Project Objectives and Outcomes in Brief 

This sub-section identifies initial project objectives (using text from the grant proposal) for the 
MCL portion of the HEA II-B grant and summarizes progress to date. 

Multnomah County Library (MCL), in partnership with the City of Portland, METRO, and other 
regional governments, will enhance citizen access and involvement with local government via the 
Internet and increase county government's capacity to utilize the Internet in its functioning. 

Specifically, MCL will: 

• Increase Regional and Local Governments Use of the Internet: Develop the Regional 
Information Technology Network (RITNet). RITNet provides a virtual place for local 
governments to meet, share information, and offer services. RITNet provides user-friendly 
citizen access to county and regional government information (including: local government 
services and personnel phone directories, county agency profiles, full-text of agency 
documents, calendars, and consumer health information) and services (including e-mail 
access to county officials, citizen-official computer conferencing via Caucus, and delivery of 
services such as licenses and park reservation forms via Internet forms) and gateway links 
to related sites (such as METRO, surrounding county governments, and the Portland visitors 
bureau). The four counties served include Multnomah, Clackamas, Washington and Clark 
(WA) and METRO (the regional governing group). In addition to those counties mentioned, 
the city governments within those counties and some non-profit organizations also 
participate. Citizen access to county government via RITNet will be supported by user 
education, distance learning, computer conferencing, and helplines. 

MCL 's RITNet met or exceeded all of its grant objectives in this area with the exception of 
calendars (in the works), e-mail (provides e-mail to content providers but many use the 
county's own e-mail for their WWW "mailto” link), computer conferencing (tried but was 
disappointing), delivery of services (presently used for job announcements and applications 
with other services in the works), and distance education (focus is on distance education at 
the regional level connected with aiding government officials building websites, see for 
example, the Community Information System Project under dissemination activities below). 

• Assist in the Integration of the Internet into County and Local Government Functioning: 
Assist adoption of Internet use by area county and local governments by providing technical 
capacity (SUN server, network links between county SAA and Novell LANs and library 
TCP networks) and Internet connection, staff development and training, technical assistance 
with web page development and data conversion. 
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MCL, supported by the grant, is a key catalyst in the coordination of information service 
delivery and the integration of Internet use in the governments of Multnomah and 
surrounding counties and with the City of Portland's Community Information Services 
Program. 

• Conduct Ongoing User Evaluation to Determine Information and Service Delivery: Adjust 
existing and initiate new information provision and services as a result of ongoing user- 
based evaluation /feedback. Develop an advisory committee from among government 
agencies. 

MCL is active in user and stakeholder evaluation and in the public presentation of 
evaluation findings via the Internet. See for example, 

Reed, Donna. (1997, February 12). Hatfield Grant-- Community information module: A 
review. Available: WWW: http://www.multnomah.lib.or.us/~donnare/report/ 

Reed, Donna. (1995 June 18 to present). Multnomah County Library Web Server Statistics. 
Available: WWW: http://www.multnomah.lib.or.us/usage/ 

Reed, Donna. We would like to hear from you: WWW feedback form. Available: WWW: 
http://www.multnomah.lib.or.us/indform.html 

RITNet is located in the Multnomah County Library and serves: Multnomah, Clackamas, Washington 
and Clark (WA) counties. Donna Reed notes (1997, February) the project assumes "that citizens' 
information needs cross geographical and political boundaries. Its purpose is to provide links to 
regional governmental information, to provide opportunities for citizen involvement and to help assure 
that there is adequate public access to the Internet in Multnomah County and throughout the region." 

Summary of Evaluator's Onsite Activities 

The Onsite Evaluator reviewed all Multnomah County Library and related web sites including 
the MCL website, RITNet (for recommendations see Appendix P-1), METRO, and a range of county 
government WWW sites. He reviewed all available documentation including the original grant 
proposal, the Evaluation Consultant's reports, and the MCL evaluation report helpfully made 
available on a web page (Reed, 1997, February 12). He prepared a preliminary draft report indicating 
current knowledge and likely onsite questions. The Onsite Evaluator met with Donna Reed, Community 
Information System Program Specialist, on June 4th and 9th 1997. Donna Reed coordinates the MCL web 
site, the county website, and RITNet. The Onsite Evaluator attended a RITNet Advisory Council 
meeting on June 19, 1997. The RITNet Advisory Council provides feedback on various RITNet activities 
and is involved in planning future endeavors. The Onsite Evaluator met on June 19, 1997 with Bill 
McCabe, webmaster of the Washington County Government website and Celia Heron, Director of the 
Office of Neighborhood Associations of the City of Portland, who heads the city's Information and 
Referral unit, and is webmaster of the City of Portland website. These individuals benefitted from 
RITNet's efforts and serve on the Advisory Council. 

The Onsite Evaluator did not meet with Jeanne Goodrich, Deputy Director and grant Program 
Administrator. Initially, she was in charge of a difficult staff reduction and transition due to 
Proposition 47 reductions in the MCL budget. Later, the Onsite Evaluator determined it was not 
necessary to meet with her because he had more than enough data provided through the above outlined 
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means. Brian Williams, Manager, Automated Systems, Multnomah County Library, who supervises 
Donna Reed on the systems side was not contacted. Again the Onsite Evaluator determined that he had 
adequate information to complete the report without further data collection. 

The MCL key liaison to the evaluation study reviewed the Onsite Evaluators findings, in the 
form of a draft report. 

Chronological Summary of Key Events and Accomplishments 

Summarized, in chronological fashion below, are key activities, milestones, outcomes, and 
accomplishments including principal work products to date given the project's objectives. Omitted are 
regular, ongoing, phone conversations or meetings at various conferences between the Evaluation 
Consultant and PORTALS staff or project participants. 

6/95 Multnomah County Library web site designed by K.C. Davenport, the library's graphic 

artist, who worked with MCL's Web Builder's Committee. 



Summer 

1995 Rhys Scholes in Multnomah County Commission Chair Beverly Stein's office conceives of 

the idea that becomes RITNet. 

9/5/95 Multnomah County website launched based on the County phone directory. This was 

Donna's first site and there was a lot of experimentation. Much time was spent positioning 
everyone on the site so that they would be pleased. A plus was that the Commission Chair 
was network savvy. For example, she did not insist that the county government be on her 
web page! 

9/95 (County) District Attorney's web site launched, among the first in county government. 

11/95 County Chair Beverly Stein's web site launched. 

11/95 RITNet Advisory Council formed and members begin meeting at Portland State University 
Library. Participants include: Regional governmental entities, Multnomah County Library, 
PORTALS, Portland State faculty, Portland Office of Neighborhood Associations, Cable 
access, members of the business community, public schools, Oregon Civic Network Coalition, 
U.S. Rep. Furse’s office, then U.S. Rep. Wyden’s office, and US West Communications. 

Three sub-committees are created: design, standards, and access /citizen involvement. 
Donna Reed, not yet hired by MCL but already designing the county’s web site, attends 
these meetings. 

The Design Committee met, designed the website using Donna’s county web site as a 
prototype, and then disbanded. 

The Standards Committee consisted of Donna, Celia Heron, and Jeanne Goodrich, Deputy 
Director /Program Administrator at MCL. They looked at existing standards (not that 
many initially) findings are incorporated ultimately into Multnomah County Web 
Standards. Available: WWW: http://www.multnomah.lib.or.us/cisp/mcstan.html which 
includes standards information used (and enforced) by MCL, RITNet, METRO and others in 
the areas of content, protocol, style, ethics/fair use, plus links to sites elsewhere. Standards 



52 




GO 



Ryan/McClure Onsite Evaluation Report: Citizen Access to Government and Other Information August 12 , 1997 



11/95 


group came to rely on local volunteer Neil McBumett, Lucent Technologies standards expert 
and founding member of BoulderNet, to supply various relevant standards as they become 
available. Celia Heron, viewing herself as non-technical, wanted preparation of clear 
explanations of the standards adopted so that non-technical people could understand what 
the standard was about and why it mattered. The process followed was that the standards 
committee would identify a standard and re-work it for the local setting, publish the 
standard on the Multnomah County Web Standards site, bring it to the RITNet advisory 
board for discussion, and then incorporate the standard (including local feedback) in the 
various web pages being designed. RITNet, the Multnomah County web page, and City of 
Portland web page (and others) all adopted standards derived from this committee's work. 
Most recent concern is meeting American Disability Act standards. 

RITNet advisory group starts a listserv (housed at PORTALS), consisting mainly of 
announcements of upcoming events, minutes of meetings, some discussion of issues 


1/3/96 


Donna Reed from the start tries to document on the Internet her knowledge base gained from 
developing local government websites for the use of new government agencies coming online 
and for use by other communities as well. Donna also found this a good way to organize her 
own thinking. See for example, Reed, Donna. Community Information System Project. 
Available: WWW: http://www.multnomah.lib.or.us/cisp/ 


1/4/96 


Donna Reed hired to start working at MCL as Community Information System Program 
Specialist (just in time for the flood!) 


1-3/96 


RITNet provides extensive, and much heralded, non-glitzy, flood information, mirrored by 
Neil McBumett at BoulderNet, which is maintained as long as the flood affects the region 
(and long after it is no longer mentioned on the nightly news). The flood occurs while MCL 
is receiving its new mainframe computer. RITNet learned how to respond to an emergency 
situation. RITNet's experience is used later as a prototype for the county emergency 
preparedness. A key lesson was that citizen’s needs are indifferent to boundaries, they cross 
government jurisdiction. 


1/10/96 


Donna meets with County Communicators group — people responsible for public 
dissemination of information within the county. In a few instances this group was helpful 
but for a variety of reasons (ignorance of the Internet, feeling that this would be a huge new 
burden put on them, no power, etc.) most did not help with bringing agencies on board. 


1-6/96 


RITNet Advisory Council at its monthly meetings spends six month period discussing the 
meaning of public access. As of 6/97 Donna still remains dissatisfied with the public access 
and interactive components of the web sites. But group consciously agreed to first build an 
infrastructure before tackling more interactive citizen involvement via the Internet. 


1996 


Donna Reed tries Harvest to index the web sites but finds, like the State Library, that 
Harvest requires someone knowledgeable in the software's intricacies (and those people 
kept leaving for other employment) 


1996 


Donna Reed starts a list for county web masters. 


1/14/96 


RITNet prototype available but not officially announced until March. 
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3/96 


RITNet officially launched with announcement at the Portland City Club (Neil McBumett 


4/96 


instrumental in getting the City Club date), two of the four counties have Web sites, City of 
Portland and Metro sites under development. 

Donna starts the County webmasters committee with webmasters from county agencies and 
elsewhere. This is largely an information sharing rather than policy group. This 
approach works initially, but over time, Donna Reed does not control who the webmasters 
are and what they do (enforcing standards becomes an issue), and it is unclear if 
communication of information presented at these meetings is being transferred from the 
webmaster to the appropriate person within the agency. For this reason, she begins, in June 
1997, working with the Operating Council and County Policy Committee on Intranets and 
the Internet. 


6/10/96 


Multnomah County Toolbox of local contacts mounted. 


6/16/96 


Multnomah County graphics bank established to address the common look-and-feel issue 
(everyone can use the same logo pulled from the graphics bank). Available: WWW: 
http://www.multnomah.lib.or.us/cisp/gifs.html 



Early Summer 

1996 City of Vancouver, WA web site launched. 



Summer 

1996 


Washington County web site launched. 


7/15/96 


Multnomah County Library, Youth Services Division and Multnomah County Health 
Department create the Reality Bytes portion of the OuterNet website containing health 
information for teenagers. Some conflict over what topics should be covered. Health 
Department offers hard hitting topics, librarians offer lightweight topics, resolution is to 
ask teen advisory board of the Health Department to decide. The teens come up with the 
best ideas. At present, each month the library youth services and health department 
alternate responsibility for what is put on the site for the month. Available: WWW: 
http://www.multnomah.lib.or.us/lib/outer/rbytes/ 


8/31/96 


Multnomah County Health Department launches its health information page. Multnomah 
County Health Department. Public health care information. Available: WWW: 
http://www.multnomah.lib.or.us/health/ed/ 


11/1/96 


Clark County’s (Vancouver, WA) web site launched. 


Fall 1996 


Clackamas County's web site launched (last of the surrounding counties). All four counties 
now have web sites available 


11/96 


The first time that the four counties election departments report election returns to RITNet. 
Previously Multnomah and Washington counties reported returns to RITNet for minor by- 
elections. These counties decide to help Clark and Clakamas counties to report their results 
to RITNet. RITNet sees its peak usage surrounding election day. For chart of usage during 
this period see: http://www.multnomah.lib.or.us/-donnare/online97/pres/elect.html 
RITNet covered on local TV, and all news media access the site. 
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11/96 Proposition 47 passes, a statewide "cut and cap" initiative that will severely affect library 
budgets. 

2/97 RITNet takes over consumer health information component from a volunteer group of 

professionals. 

2/12-19/9 7 Donna Reed produces her evaluation report requested by the Evaluation Consultant and 

made it available on the Internet <http://www.multnomah.lib.or.us/-donnare/report/>. 

May 1997 Environmental information category added to RITNet. 

6/97 Donna Reed starts meeting regularly with the County Operating Council (Deputy Director 

level of county government) in a two week period Donna added three new agencies to the 
county web not previously willing to participate. 

Spring 1997 Donna Reed becomes a Member, Oregon Telecommunications Forum Council (OTFC), 

Community Development Team which made a recommendation to the governor and state 
legislature for a state community networking model which is awaiting action. 

Spring 1997 Donna Reed becomes a Member, Internet/ Intranet Oversight Committee which is 

subsequently changed to the Internet Technologies sub-committee of the Information 
Technology Council (ITC). 

Spring 1997 Donna Reed becomes a Member, County Information Technology Council (ITC). 

6/97 The County commission presently recommends, but does not mandate, that its agencies have 

web pages. Donna Reed believes this may change in the near future due to the rapid 
progress made by the City of Portland (where agency web pages are now mandated) which 
at first was behind the county but now is catching up fast. 

Staff Development Activities 

This sub-section summarizes efforts by MCL to obtain education and training for staff to better 
implement the grant objectives. 

Donna received her M.I.S. during this period from Emporia State University’s (Kansas) 
distance education program (as did Sue Jenkins at Driftwood Library, Lincoln City). She took two 
specifically relevant courses on information design as part of her Masters work at Emporia State. 

When Donna needs to learn a particular piece of software (e.g., Photoshop, web editors, Windows 95) 
or technology she goes to local vendor contacts and has them train her in the software’s use. If the 
product is brand new she tends to go to someone local whose software or technology skills she respects 
and "picks their brain." She relies heavily on the Powell’s Technical Bookstore (one of the best in the 
country). She attended various professional association (ALA, PLA, ASIS) workshops (particularly on 
information and communication design aspects, remembering Edward Tufte’s workshop) and attended 
national conferences on community information. Donna also used the Internet (and phone) heavily, 
identifying interesting projects around the country, and contacting that person directly. 
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One way of viewing most of what Donna Reed does is as staff development for county and 
regional governments connected with the integration of the Internet into government. When Donna first 
started, she offered classes on various levels of HTML. All HTML 101 class members finished by 
producing a web page. But Donna Reed stopped offering HTML classes when other sources such as the 
local colleges, started offering HTML courses. But Donna worries that these courses do not emphasize 
standards and people do not know how to read their own code because they are taught using an HTML 
editor only. Donna also noticed that if people do not use HTML right after the class they lose it. Also 
if agencies don't pay for the course, they don't value it. 

Project Work Products 

This sub-section lists and describes the major products, services, and other outcomes as a result of 
the grant: 

Multnomah County website. Available: WWW: http://www.multnomah.lib.or.us/ 

Multnomah County graphics bank. (1996, June 12). Available: WWW: 
http://www.multnomah.lib.or.us/cisp/gifs.html 

Multnomah County Health Department (MCHD) site. (1996, August 31). Available: WWW: 
http://www.multnomah.lib.or.us/health/index.html 

Multnomah County Library. Available: WWW: http://www.multnomah.lib.or.us/lib/ 

Multnomah County Library, Youth Services Division and Multnomah County Health Department. 

(1996, July 15). Multnomah County Library YA Web Page: OuterNet, Reality Bytes. Available: WWW: 
http: / / w w w .multnomah . lib . or . us / lib / outer / rby tes / 

A young adult WWW site containing consumer medical information. 

Reed, Donna. Community Information System Project. Available: WWW: 
http://www.multnomah.lib.or.us/cisp/ 

Reed, Donna. Community information system projects: A presentation given to the Multnomah County 
Commissioners. Available: WWW: http://www.multnomah.lib.or.us/~donnare/cispres/ 

Reed, Donna. (1997). Community information system projects sponsored by Multnomah County Library. 
Portland, OR: Online Northwest 97. Handout. Available: WWW: 
http://www.multnomah.lib.or.us/Donnare/online97/ 

Reed, Donna. Community Information System Project: A toolbox for people working on RITNet and 
Multnomah County Web sites. Available: WWW: http://www.multnomah.lib.or.us/cisp/ 

Reed, Donna. (1995 June 18 to present). Multnomah County Library Web Server Statistics. Available: 
WWW: http://www.multnomah.lib.or.us/usage/ 

Reed, Donna. What is the Community Information System Project? Available: WWW: 
http://www.multnomah.lib.or.us/cisp/cisp.html 
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RITNet. Available: WWW: http://www.region.portland.or.us/ 

Consists of: 

• A website for each County agency describing that agency and its work. 

• Public information, such as pamphlets and educational materials, produced by County 
agencies 

• Interactivity in various formats- forms , "mailto" links, conferencing and opportunities for 
citizen involvement. 

• Links to externally-created sites containing information of interest to those living in the 
county and to those wishing to visit or do business in the county. 

• Interactive, continuously changing information, that is driven by customer needs. 

Dissemination of Project Results 

This sub-section identifies specific ways MCL disseminated project findings to the local, state, 
and national communities and the profession including publications, videos, press releases, 
presentations, external training sessions, etc. 

Community Information System Project 

Reed, Donna. Community Information System Project. Portland, OR: Multnomah County Library. 
Available: WWW: http://www.multnomah.lib.or.us/cisp/ 

Donna Reed maintains the above web page to assist those who need initial help in community 
information and web page creation. Included on the site are the following areas: About the Community 
Information System Project (CISP), A Toolbox for Multnomah County Web builders (Before you begin , 
Frequently asked questions (coming soon), Getting an account, Multnomah County Web Standards, 
Beginning HTML, Graphics 101, Multnomah County Graphics Bank and Related Sites), RITNet Toolbox 
and Neighborhood Associations links. Donna developed the site at first to aid local county and city 
governments who asked Donna for help. The principal being: you want to use the web, well use the web 
to obtain help. The site gets much wider use now. 

Local, Regional, National Conferences 

Donna Reed speaks at numerous local (including all of the surrounding county governments and 
all of the city governments in Multnomah county), regional, and national events on various aspects of 
providing local government information via the Internet. Often putting her presentations up on a web 
site, for example: 

Reed, Donna. Community information system projects: A presentation given to the Multnomah County 
Commissioners. Available: WWW: http:/ / www.multnomah.lib.or.us/~donnare/cispres/ 

Reed, Donna. (1997). Community information system projects sponsored by Multnomah County Library. 
Portland, OR: Online Northwest 97. Handout Available: WWW: 
http:/ / www.multnomah.lib.or.us/Donnare/online97 / 

RITNet Listserv 

PORTALS houses a listserv for RITNet members. 
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Evaluation Efforts 

This section identifies specific ways MCL used to evaluate the project. 

Donna evaluated web pages using interactive forms 4 on each site soliciting feedback. For an 
example of a feedback form see: 

Reed, Donna. We would like to hear from you: WWW feedback form. Available: WWW: 
http://www.multnomah.lib.or.us/indform.html 

Questions asked included: Was service provided appropriate to your needs? Was the service easy to 
use? Was the information provided current? Useful (why)? Did you make use of any help desk 
services? Training? Was the help appropriate, useful? Overall suggestions to improve the service? 
There are about 10 feedback comments a day. Donna tries to respond within 24 hours. As the various 
county agencies take over managing their own information on the web site the feedback is directed to an 
appropriate person in each agency. 

Donna also tried log analyses of the various web pages. While log analysis is not particularly 
accurate Donna finds log analysis useful for suggesting trends and has not yet written off its utility. For 
RITNet log analysis statistics see: 

Reed, Donna. (1995 June 18 to present). Multnomah County Library Web Server Statistics. Available: 
WWW: http://www.multnomah.lib.or.us/usage/ 

The monthly RITNet Advisory Council (and supporting listserv) are important sources of 
evaluative information. 

Donna s evaluation report for the HEA II-B grant was unique because of its availability on the 
Internet: 

Reed, Donna. (1997, February 12). Hatfield Grant- Community information module: A review. 
Available: WWW: http://www.multnomah.lib.or.us/-donnare/report/ 

This helped the Evaluation Consultant who could access the report while traveling This method of 
publicly providing evaluations may have merit in other circumstances as well. 

A principal form of evaluation for Donna is direct contact with various interest groups connected 
with county government — even talking to people on the bus! Donna regularly meets with the North 
Portland Neighborhood Coalition, located in a low income area, to try to find meaningful ways that 
people with low or no budgets can participate on the web. One useful source is to talk with key service 
points within local governments to find out if the web sites have an impact in addition to finding out 



4 The interactive forms used by RITnet were originally questions about specific parts of the 
page, whether the user got the information they requested, or simply if you have any comments click 
here . The user would then e-mail RITNet using the mailto option described in the PORTALS section. 
More recently, RITNet uses simple CGI forms which can look more like a traditional survey, can be used 
at public terminals (like those in libraries), and do not require e-mail accounts. 
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the latest frequently asked questions. Another useful source is the local alternative and free 
newspapers for current hot topics. 

Donna is actively involved in the standards setting process beginning with the standards 
committee of the RITNet Advisory Council. Donna regularly checks to see if agencies use the standards 
and once even temporarily removed an agency from the website due to violation of the standards. 

Lessons Learned 

This sub-section identifies significant lessons learned by MCL which may also be of interest to 
similar organizations in other settings. 

Strategies for connecting a county to the Internet 

Three useful approaches tried in parallel with synergistic outcomes worked to initially 
acquaint local governments to the Internet: opportunity/emergency, frequently asked questions at key 
service points, and approaches to individual agencies. 

Donna started work with a baptism by water — a flood emergency affecting the entire region. 
Donna began identifying needed information and information providers across the various governments. 
She acquired this information often via phone and fax, and then entered the information on to the 
county web page. The emergency situation enabled government agencies across jurisdictions to cooperate, 
particularly with the library, a perceived neutral player. The success of Donna's effort here (1) gave 
her a product to show the ignorant or skeptical (2) an entre into further collaboration with the agencies 
who had contributed to the flood emergency effort, and (3) word-of-mouth recognition when working 
with new agencies. 

Donna also focused on identifying the information most needed/used by citizens. She mentions 
that the flood taught her "how to listen for what citizens needed and where to look for it." She did 
this by talking to the key services points within government: the information and referral unit, the 
reference desk of the public library, and receptionists and secretaries at the government agencies. 
Agencies saw the importance of getting out this key information any way possible — even on the 
Internet! Internet users began to find the information they needed available on the Internet. Agencies 
saw their information on the Internet and began to get requests from people who had learned about the 
agency via the Internet. Suddenly the agency had a stake in what was on the Internet about them. 

The third parallel strategy involved group and individual contacts with government agencies. 
Donna said the second thing the flood emergency taught her was how to approach agency officials. 
Donna addressed the county communicators group, technologists, deputy directors, and others. Donna 
also worked with individual agencies as interest in participation emerged. 

Donna and now county government officials, select information sources using a number of criteria 
including: was the information needed by citizens (as determined by their requests to government 
service points); was the information available in digital if not HTML format, and could updated 
information be easily obtained. 

When Donna started in January 1996 her approach was to solicit the information from the 
agencies and then key it on to the website using HTML coding. The agencies did not know about the 
Internet and much of their information was in paper-based format. Over the eighteen month period 
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this dramatically changed. Many agencies have their information now in digital format, and agencies 
are now responsible for coding and maintaining their own portions of their Internet presence. Another 
welcome change is the gradual shift from Donna seeking out information, to agencies coming to Donna 
with the information to be placed on the website, to agencies maintaining their own data on the 
website. 

For Donna, the progression was from keying in paper based information from an agency, 
establishing a mechanism for regularly obtaining current (and new) information from an agency, 
obtaining the information in digital format, and finally getting an agency to manage its own portion of 
the government web page. On the other hand, for the user, the look-and-feel from the beginning is of a 
fully functioning county government on the Internet. This approach allows agencies with different 
learning curves, priorities, resources, and skills to move more at their own pace. 

Proposing an Agency Website: Who Offers Assistance and Who You Talk to Matters 

Who do you ask when you want to involve an agency with the Internet? Do you start with the 
receptionist and let them choose? Do you ask for the technology or systems unit? Do you start with the 
agency head, or somewhere else? Donna, in learning how to assist an agency with its Internet use tried 
all of these approaches with varying results: 

• Receptionists and secretaries frequently know what the public considers the most important 
information that an agency produces ~ because they are constantly asked for it. 

• Donna was often handed off to the technology unit within an agency because "that is what 
the Internet has to do with, right?" The technology unit was useful for identifying someone 
within the agency who could do any HTML coding or electronic data transfer to the web 
site. But the technology unit did not know the information of value, could not guarantee the 
flow of an agency's information to RITNet, nor did it have the power to authorize an 
agency's participation on the web. 

• The agency communicator's branch (public relations or marketing) often had the best 
presented least useful information the agency offered. The communicator's were often not 
technologically savvy and, as a result, did not readily grasp the significance of the Internet 
for their work. 

• Agency directors were initially hard to reach and uninformed about the Internet. 

Donna had her most recent successes addressing the agency deputy director level — smart enough to 
know the Internet's utility, knowledgeable enough about the agency's products and services to prioritize 
their importance for presentation on the Internet, and powerful enough to make it happen within the 
agency. 



Who you are (the role you play) and the agency you represent matters when you offer to 
provide assistance in utilizing the Internet. The library is seen as information neutral, that is, the 
library is interested in acquiring and presenting information an agency might have without adding a 
library spin to the data. This meant that agencies were more ready to talk to a librarian than others. 
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Steps to Establishing an Agency Website 

Today, Donna would begin her contacts with an agency new to the web at the Deputy Director 
level. She would have that person organize a meeting of the people the deputy thought should be 
involved. Donna would suggest that department heads be at the meeting, possibly some clerical staff 
(if they are the ones answering the frequently asked questions), and a representative from the 
information services component. Donna stresses that it is important to have an information services 
person on board but that their role is to support not control the project. At the meeting, Donna would try 
to gain an understanding of how the agency supplies information to the public. This involves looking at 
information delivery, who does what informational task. The mechanism varies from agency to agency 
often around two approaches. One model is a department that supports internal customers -- county 
employees. Another model is a department that directly serves the public. 

Donna then tries to put a model in place that fits in with how the agency already processes 
information. In that way the creation of information for a web page will fit right into existing practice. 
Moving the creation of the web page away from whomever originally creates the information within 
the agency can be a real problem. Often the person in charge of creating the information is in charge for 
a reason (not apparent to the outside consultant) and the (apparent) loss of control (even if it is copying 
exactly what the original person created) is resisted. Often one unit within a department is unaware of 
information being created in another unit, and the information unit in any department is regularly the 
last to know in the agency! Thus having every unit involved and in control using existing information 
processing patterns is the best approach. This is not to say that opportunities for re-engineering 
workflow are not possible. But one should be cautious in changing something not fully understood in the 
time available. 

The model that Donna initially chooses for an agency will depend on whether or not she senses 
that the department (or key members in it) is intimidated by using the Internet or reluctant for a range 
of other reasons (e.g., fear of added workload, ignorance, etc.). If she senses resistance she will start 
with a simple task. She looks for obvious department information products, or a crisis line, or 
brochures. A simple place to start is to have one or more people send a fax, or more advanced, a disk of 
information one time, or better still on a regular basis, to Donna. Donna will then encode the 
information and put it on a web page. Donna will then return to the department to show the original 
information now on the web. The purpose is to get feedback, but also to make the web seem more 
familiar. People seeing work with which they are familiar in a new medium suddenly do not find the 
medium so strange. Later, as the department feels more comfortable with the Internet other tasks, 
including internal production of web content, become possible. 

The model that Donna initially chooses for an agency also will depend on the type of support 
available within the agency. What inhouse expertise is available? Can that expertise be tapped? 
Sometimes there will be a person who knows HTML but should not have access to the information, or 
the person’s salary ranking is too high or two low, or the person must cross some other organizational 
barrier to contribute what is needed. What expertise does the department that Donna can offer? If 
departmental expertise is available, a critical period in the process begins -- moving the agency to take 
control of its own web information. 

A key point in the relationship is getting the agency to take control of its own web information. 
How did Donna encourage this process? Related challenges are: How does the county government 
control for quality and encourage a common look-and-feel yet not dampen enthusiasm? To get a 
department web site up Donna will go to great lengths (spending several weeks eight hours a day on 
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site)! Donna will often print out various standards and bring them to a department to work with the 
departmental expert and constantly refer to them. Donna will offer to create a template page to start if 
asked (the CISP site below has some templates already on it). But she is not shy about directing 
people to other experts if appropriate. Several elements are key: Donna's approachability, her active 
presence on the Internet locally and in person around local governments including the willingness to take 
as long as it takes on site at key moments in the process, and the web-based resources Donna makes 
available for new government agencies wanting to establish their agency’s Internet presence. 

Reed, Donna. Community Information System Project. Portland, OR: Multnomah County Library. 
Available: WWW: http://www.multnomah.lib.or.us/cisp/ 

This site provides basic introductions to the Internet tools a web site builder would need, an account to 
get started, an explanation of key standards and why they matter, a graphics bank containing 
frequently used county symbols (addressing the common look-and-feel question), and a list of key 
contacts who could be called on for help. Simple, but effective, not only within Multnomah County but 
at the municipal and regional levels of government as well. After a web page is launched Donna may 
help with organization and she regularly checks to see if the page is being maintained (and if not, why 
not). 



From the beginning, Donna, had as an end goal, the provision of department services over the 
Internet. A service, in Donna’s terms, is some type of interactive exchange on the Internet: a forms based 
request by citizens and response by an agency (or vice versa), or some type of database searching. Donna 
notes that the way to find out about the services is to find out who is creating information about the 
services and following the information about the service around the department. The delivery of a 
service usually involves the need for a more technical person (e.g., CGI programmer, database person). 
This is where the library is now - the state of the art. The library hopes to be offering database 
services on a regular basis by the end of the year. 

Services, like database access, need to involve more people higher up in the agency before being 
released to the public. Departments are fearful that citizens will use the data available in a database 
to embarrass the agency. Several instances of this occurred already making a cautious department even 
more cautious. So every possible of use of the data and every use of the data matched with data from 
other sources must be considered. This is a slow process at best. Further, there is the likely possibility 
of citizens mis-interpreting information contained within the databases. The possibility of mis- 
interpretation puts the burden on government public information personnel to explain increasingly 
complex information problems before the local newspaper reduces the issue to a nightmare. 

Glitzy, Bureaucratically Correct, or Useful Information Content 

Providing information on the flood taught Donna several lessons including: 

• There is a difference between a website designed for immediate impact and ’’glitzy” appeal 
and a site which seeks to convey information which, in this case, flood victims and local 
government officials need to assist in the recovery process. Related is the idea that the 
glitzy sites lose interest when the topic is no longer newsworthy, but the need for emergency 
information may persist for months after the initial disaster. 
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• Citizens needs for government information, particularly in a crisis, cross government 
boundaries. Indeed, citizens neither know or care whether the information they need is 
city, county, state or federal government information. They need the information and need 
it now! This argues in Donna's mind for an information service in which governmental 
jurisdiction is transparent to the users. 

Donna further learned about the importance of the clear, orderly presentation of information when 
working on making her web sites useful to those who were blind. What looked excellent with great 
graphics and layout to the sighted appeared disorderly to the visually impaired. 

When comparing her sites to others Donna learned the importance of accuracy in the titles and 
descriptions of sources. Other sites frequently mis-label or mis-represent Donna's web sites. For 
example, mis-naming the Multnomah County web site as the Multnomah County Library web site. The 
need for accuracy by constantly checking links also matters to users. There are regular conflicts 
regarding authority and origin of information on the web. Too often, a web site will give the impression 
that it creates all of the information presented at the site when much of the site merely contains links 
to information sources created by others: a form of web plagiarism perhaps. 

In summary glitzy is nice, Donna will be working with a graphic designer this summer to re- 
design several of her web pages. However, there is no substitute for accurate, useful information that is 
well organized and presented. This also pointed up to Donna the numerous problems created by well 
meaning people who were not trained in basic library/information skills. 

Bonus: Making the Unexpected Inter-Governmental Connections 

Donna regularly discovers links between government agencies that the government agencies 
themselves do not know. For example, in preparation for making a domestic violence pamphlet 
available on the web she learned that the Children and Family Department had a Domestic Violence 
Unit, the Health Department had a Domestic Violence Unit, the District Attorney's office has a 
Domestic Violence program and the City of Portland.... Donna forged both virtual and real links. 

Internet Services are for the Elite Few: Why Should Agencies Bother and Citizens Fund? 

Today not everyone has an Internet connection despite dramatic growth in use. There is great 
concern that the economic and social have-nots of the present will not be connected to the Internet so as 
to have an equal or greater chance at a better future. Those with these concerns see the Internet as a toy 
for the elite few rather than able to contribute to the benefit of the many, in particular the have-nots. 

If this is the case, why should agencies bother to connect to the Internet and provide information and 
services via the Internet? Donna Reed's experience suggested two potential reasons why delivery of 
information and services via the Internet should matter to county government agencies. First, Donna's 
experience with the flood emergency eighteen months ago taught her that those already connected to 
the Internet include community opinion leaders, county government officials, public information 
officers, libraries, key community organizations, the media, and other disseminators of information. 
Donna obtained information vital to the flood emergency from the Internet, and presented vital 
information to the above key community members which made a difference. Second, Donna, who 
considers herself knowledgeable about who uses the network for what, says that she is constantly 
surprised who in actuality have access to and really use the Internet. It may be that Internet use is not 
being determined by economic and social advantage alone. Today there is a dramatic increase in the 
number of individual users and in public provision of Internet services in the METRO region. 
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Next Steps 

This sub-section suggests next steps to be taken as this grant period comes to an end and to 
, advance the project s objectives after the funding period. Donna Reed identified the following next 
steps during the interviews with the Onsite Evaluator: 

• Add new information products and services to web sites: Already underway are plans for a 
professional graphic artist to re-designing several of the sites, there will be new sections on 
community health, the environment, the non-profit sector, and an emergency preparedness 
site on RITNet sponsored by a group of regional governments. Planned are more interactive 
services including citizen access to departmental databases, the addition of maps to web 
pages, and plans for a global index of county government web sites. 

• Embed electronic networking in local governments: There are active discussions among local 
governments about the benefits of mutual web site development. For example, there are 
discussions between Multnomah County and City of Portland on shared, web based, 
information and referral. Government intranets will take hold in the next year and, as a 
result, individual departments will have increased incentive to digitize and subsequently 
network information. The process of embedding electronic networking within the county 
government is an active focus of the Internet Technology Council. Issues being considered 
include development of departmental technology plans, modification of job descriptions to 
reflect changes in roles brought about by the Internet (and other information technologies), 
the relationship and communication between county and departmental information services 
units, hiring and reward of personnel with newly valued information technology skills, and 
union consequences. 

• Agency and government level policy issues will receive intense scrutiny: Intranets, Internets, 
Extranets; citizen interactive access to core agency information (with services not far 
behind) all force senior management involvement. Policy questions include: Who owns 
government information? Who, under what conditions, should have access to government 
information? The impact of telecommuting of government and citizens? Should certain 
agency information now be required to be on the Internet (e.g., organizational structure with 
description and contact information) and maintained up to date? What should be the 
relationship between the county information services department, individual department 
information services units, the individual creator/producers of information, and die users of 
that information? For example, in the area of web page standards? 

• Public access at the neighborhood level: Many of the original RITNet members felt that 
the Internet represented an opportunity to improve public access to the least served. Has 
it? A continued effort will be made to extend the public Internet access to neighborhood 
coalition organizations. 

• RitNet market strategy: Donna Reed believes the RITNet story needs better telling, she 
suggests there is a need for a market strategy for RITNet. 

In addition, planning is underway for a fall governments and the net conference to be held at METRO. 
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Recommendations 

The evaluators offer the following recommendations based on the onsite visit and subsequent 
discussions. 

The Multnomah County Library portion of the HEA II-B grant was a clear success with all grant 
objectives met or exceeded. The library is clearly a valued player in city, county, and regional 
government use of the Internet to provide government information and services to citizens. RITNet 
process and products can serve as a useful model for other local governments seeking to integrate the 
Internet into government activities. Several recommendations occur to the evaluators: 

• Disburse monies committed to MCL: Christina Dunn, Director, Department of Education 
Discretionary Programs, Library Programs Office in a letter received by PORTALS on 
March 27, 1997, responding to a March 25, 1997 letter from Howard McGinn, then PORTALS 
Executive Director, noted: "You may grant funds to support the Multnomah County Library's 
RITNET; the $50,000 cited seems reasonable for this activity." To date, MCL has not 
received these funds. The evaluators recommend this money be promptly allocated to MCL. 

• Get the word out: There is no local reward for documenting the County and RITNet 
experience for use by other libraries and local governments. Yet the need and utility of 
Multnomah County's experience within the state, the region, and nation is great. The 
evaluators urge the participants in this portion of the grant to get their message out! Focus 
on capturing the voices of experience who can discuss not only what was done (and what was 
not) but why? It might be helpful for the grant principals to visit other settings to gain 
perspective on how much they have to offer the professions involved. The evaluators 
applaud Donna Reed's (and no doubt others) participation in regional and state wide 
committees and the American Library Association. 

• Address anew fundamental value questions regarding electronic networking's worth to 
governments and citizens: The RITNet Advisory Council began with strong concerns about 
public access but decided to build infrastructure first. The Internet Technology Council 
exists and has begun to ask the tough policy questions connected with embedding electronic 
networking in local government. There are active discussions regarding partnering between 
county and city governments in the provision of information and services via the Internet. 

To the outside evaluators, these events suggests that a period of time will be needed to re- 
assess and re-address fundamental value questions regarding electronic networking's worth 
to governments and citizens. This re-assessment is both necessary and worthwhile because 
there is a critical mass of experienced government personnel that can begin to derive 
meaningful answers. This was not the case eighteen months ago when the Internet's 
potential was new to all. 

The evaluators suggest that it is time to bring together experienced government personnel to 
re-ask such questions as: Can the Internet increase citizen public access and interaction with 
governments? Is this a good thing - for citizens, for governments? What exactly does 
public access mean in a digital era? Has the Internet any value for poor, urban, 
neighborhoods? Have these neighborhoods been forgotten? Is there a list of basic 
information products and services that all governments can and ought to be offering their 
citizens via the Internet? Why? In sum, all of the issues previously deferred to build 
infrastructure and to gain experience, will be back. These issues deserve fresh attention, not 
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so much because they will be solved, but to aid in the consolidation of grant participants 
learning to date, and because the answers derived now will be different and better for the 
communities served. 

• Establish relationships with additional key user groups: MCLs approaches to getting 
agencies on the web include taking advantage of opportunity/emergency (e.g., flood, 
election), or subject/ information producer interest (e.g., health information), or responding 
to early-adopter agencies. An alternative approach might be to think of a group of users 
with which local government needs to have a special relationship. Then in dialog with 
that group of users, create a networked service which encourages the relationship needed. 
An additional criterion would be whether, or what type of network connection does the 
targeted group of users have. Several groups come to mind. Schools and school children: 
What does county government want every elementary school child to know? What does 
every elementary school child want to know about county government? If nothing occurs to 
you find a couple of creative elementary school teachers to come up with a unit on You and 
your County! Nursing homes and Senior centers: In what ways do (could) Seniors and county 
government interact say in health, benefits, taxes? This might mean getting away from 
sources of information, getting away from government services, and talking to groups of users 
in their setting. But it might be an another way of learning how best citizens and 
governments can electronically network together. 

• Continue the process of embedding electronic networking in government: As in any project, 
the last mile is the toughest (and least gratifying). After a wonderful beginning, it is time 
to move to the endgame with the same patient enthusiasm as brought to the start. 

• Explore the services side of the picture: MCL s goal from the start was the provision of 
information and networked services. MCL experience helps identify the issues and 
difficulties involved both technical, economic, and social/political. The next logical step 
is to gain experience with electronic service delivery as you gained experience with 
electronic information delivery. As a start, the possibilities and consequences of two-way, 
non-financial, government-citizen exchange wait to be explored. 

In an experimental area, such as the one chosen by Multnomah County Library (and partners), the 
library's success in its project activities is outstanding. The library should be congratulated for their 
efforts and successes. 
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V. Oregon Historical Society (OHS) 



This section of the report examines the Oregon Historical Society’s programmatic component of 
the HEA II-B grant including: program objectives using language from the original grant proposal and 
outcomes to date in brief, the evaluator’s activities in order to compile this section of the report, a 
chronological summary of key events and accomplishments, staff development activities, project work 
products, efforts to disseminate project results, Oregon Historical Society’s evaluation efforts, lessons 
learned, next steps, and the evaluators’ recommendations. 

Project Objectives and Outcomes in Brief 

This sub-section identifies initial project objectives (using text from the grant proposal) for the 
OHS portion of the HEA II-B grant and summarizes progress to date. 

Oregon Historical Society will make portions of its collection of Oregon historical materials 
available by conversion to digital form, initial organization, and provision of access to Oregon citizens 
via the Internet. Specifically, OHS will: 

• Produce a series of electronic educational packages for grades 4-14: the Oregon State 
Archives was the sub-contractor for this effort. Packages to be available on the Internet 
include text of historical documents with interpretive notes of appropriate reading level. 

There were twenty three items in the Echoes of Oregon History set as of May 15th 1997. See 
Oregon State Archives. Echoes of Oregon history, 1837-1859. Available: WWW: 
http://arcweb.sos.or.gov/echoes/defaultechoes.html 

• Digital selected items from OHS collections: Digitize 10,000 historical items from the OHS 
collection including 2500 manuscript pages, 300 maps (in image not in GIS format), 2500 
photographs, 100 oral histories (and sound recordings) of central government figures (e.g., 
state legislators), and 5,000 items from the artifact collection. Digitizing process includes 
selection of materials from existing collections, production of slides, conversion to Photo CD, 
and then conversion to hard disk storage. 

OHS made substantial progress in this area, for further details see the work products 
section below. 

• Digitize and expand finding aids to OHS collections: OHS planned to catalog the 
individual manuscript page digitized and the parent collection from which the page 
comes, digitized maps, the digitized photo and its parent collection, the oral histories 
chosen for digitalization and add descriptive information about the artifacts digitized. In 
addition OHS planned to add local serial holdings data to OCLC (and OHS's Horizon 
catalog). 

Recently, as a result of the Berkeley Finding Aid Project (and others) OHS also planned to 
prepare, update, or convert to Access databases finding aids (called variously finding aids, 
indexes, databases, inventories) for map collections, oral histories, manuscripts, images, 
and artifacts. In general, these databases link descriptive information about the item in 
the OHS collection, the location of the item, and the location of the digital representation 
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of the item where available. OHS plans to index the catalog and all of digital finding 
aids into one (meta) database accessible to the Internet using Inquery software. 

OHS made substantial progress in the creation and conversion of finding aids, for further 
details see the work products section below. The Inquery software to create the meta index 
is about to purchased and data structures are ready for it. For further information on the 
Inquery software see: <http://hobart.cs.umass.edu/public.html>. 

• Website Development, Establish a virtual OHS on the Internet: Make above digitized 
items including finding aids, the finding aids meta-index, digital representations of 
selected items from the OHS collection and the State Archives electronic education 
packages available via a website on the Internet. 

OHS introduced a website which includes digital representations of the collections at 
OHS, finding aids, and other information about OHS. Available: WWW 
http: / / www.ohs.org/ 

• Conduct related user education and staff development: OHS as an institution moved from a 
paper-based environment to a digital environment in part as the result of this grant. OHS 
staff and users needed training to take advantage of the new environment. 

OHS staff attended a range of staff development activities connected with the grant. OHS 
has not begun user education activities related to the website or grant yet. 

• Conduct continuous evaluation of project and user feedback to it: Evaluation will include 
collection of benchmark data on portions of collection digitized, monitoring of staff hours 
spent on the project, and user-based evaluation of resulting project products and services. 

OHS evaluation efforts are considerable, ongoing, and well done. OHS should assess the 
users of its website, their needs, and their interest/ ability to contribute to OHS. 

These activities will make OHS and its collection a much stronger and accessible presence within the 
state and nationally. An OHS patron will not always need to be physically present to use finding aids 
and the collection. OHS will increase its ability to organize and present its collections in novel and 
imaginative ways. These grant activities are one element in OHS’s strategic plan presented in 
Appendix H-l, "Oregon Historical Society: Strategies through 2005." Appendix H-2 presents the OHS 
organizational chart. 

Summary of Evaluator’s Onsite Activities 

The Onsite Evaluator examined existing available documentation on the OHS project including 
the original grant proposal and the evaluation report filed by Todd Welch (1997, January 31). The 
Onsite Evaluator reviewed OSL website, for recommendations see Appendix P-1. 

On June 5, 1997 the Onsite Evaluator met with the majority of OHS staff involved with the 
project in a group introduction and discussion. Then spent the rest of the day meeting with Elizabeth 
Winroth, Maps Librarian, Laura Ayling, Map cataloger, Jim Labosier, Maps Assistant, and Mandy 
York, Maps and Photographs Cataloger; Barbara Abrams, Project Navigator and Deputy Director for 
Operations and Programs; and, Steve Hallberg, chief cataloger. 
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On June 10, 1997 the Onsite Evaluator spent the day at OHS meeting with Kris White, Director 
of Archival Collections; then Marsha Williams, Director of Museum Collections; then Chris Bostic, 
MIS Director and Communications Manager and Dwight Patterson, MIS Senior Engineer ; then James 
Strassmaier, Oral Historian; then Sue Seyl, Director of Image Collections, Richard Yost, Imaging 
Technician, Mandy York, maps and photographs cataloger, Elizabeth Winroth, Maps Librarian, and 
Mikki Tint, Assistant Photographs Librarian, Evan Schneider, Photographer all from the 
Photography and Image Department. 

On June 11, 1997 the Onsite Evaluator conducted an extended meeting with Todd Welch, Project 
Archivist and met with Chet Orloff the Oregon Historical Society Executive Director. 

On June 12, 1997 the Onsite Evaluator examined minutes from the OHS Web Task Force and 
other documents connected with the project. 

Due to scheduling difficulties the Onsite Evaluator did not meet the person directly responsible 
for development of the "Echoes of Oregon History "series at the State Archives. The Onsite Evaluator 
did meet with Roy C. Tumbaugh, the State Archivist, on June 6, 1997 in Salem, Oregon however. 

Chronological Summary of Key Events and Accomplishments 

Summarized, in chronological fashion below, are key activities, milestones, outcomes, and 
accomplishments including principal work products to date given the project's objectives. Omitted are 
regular, ongoing, phone conversations or meetings at various conferences between the Evaluation 
Consultant and PORTALS staff or project participants. 

1975 Photography Department receives North West Area Foundation grant to reorganize the 

photography collection, this marks the beginning of computer use in the department. 

1979 Photography Department receives NEA grant for Cronise collection cataloging, exhibition, 

and book, involved computer use for cataloging 

1979 Photography Department receives NEA grant for data entry 

1984-86 Photography Department receives NHPRC grant to complete implementation of Pre- 
MARC computer cataloging scheme for photographs using a Wang VS80 

1995 Neighborhood Windows project for Portland exhibit provides Photography Department 

with good experience with imaging. 

1-9/95 OHS prepares HEA II-B grant proposal. Grant team includes John Mead, Kris White, and 
Sue Seyl. 

9/95 OHS, using funds from the Murdoch Foundation, makes public access Horizon catalog 

available. 

10/1/95 HEA II-B awarded to PORTALS, OHS estimates it takes six months for PORTALS, OHS, 
and the Department of Education to agree on terms of reporting mechanisms, financial 
disbursements, contractual obligations, and scope of work. 
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10/95 

10/95 

11/1/95 

11/30/95 

11/95 

12/19/95 

12/95- 

1/15/96 

12/26/95 



12/95 
to date 

1/96 

1/96 

1/96 

1/30/96 

2/96 

2-3/96 

2/21/96 

3/5/% 



Argus artifact database contains slightly over 20,000 artifact descriptions at the beginning 
of the grant period. 

Todd Welch, as Secretary /Treasurer of the Northwest Archivists, creates their web page. 
At the same time he prototypes an OHS web page. 

Charles McClure, Evaluation Consultant, meets grant principals in Portland. 

RFP released for consultant, responses due by 12/20/95 

Todd Welch demonstrates prototype OHS web page to Sue Seyl and Chris Bostic 
Charles McClure, evaluation consultant conducts conference call with grant principals. 



Began telephoning consultant candidates asking them specific questions OHS needed 
answers to at that point in time. This turned out to be a useful strategy for narrowing the 
field. 

OHS, PORTALS, and Department of Education sign contract back dated to 12/1/96 to begin 
the project (Chris Bostic and Todd Welch negotiate for OHS). Then the federal government 
shut down. Discussions begin internally regarding how much money is available, how 
money would be allocated for staff, supervision, technology, software, training, and 
consultants. Contract required that OHS hire a consultant to help with the project (OHS 
eventually hires two beginning 1/30/96). This was viewed as a good idea by OHS as they 
felt they "were in over their heads." 



John Mead, Director of Reference and Research Collections, begins biweekly collection of 
benchmark data (synchronized with pay periods) on portions of collection digitized 

OHS continues search for consultants, find no one person that suits but two, Beth Sandore 
and James R. Blackaby together, have what they need. 

OHS holds an all staff computer conference. 

OHS concludes that a Gopher site will not do for its digitized items, begins to consider 
WWW. 

Beth Sandore and James R. Blackaby hired as consultants. 

Purchase of microcomputers associated with the project. 

OHS employee Internet access begins. 

Todd Welch demonstrates mock up of OHS web page. 

OHS staff conference on automated collection management in the Madison room, the focus is 
on the HEA II-B grant 3-8 PM 



Ryan/McClure Onsite Evaluation Report: Citizen Access to Government and Other Information August 12, 1997 



3/5/% 


Web Site Task Force formed by Executive Director, Chet Orloff around same time as 


3/14/% 


consultants visit to plan for incorporation resources from all OHS departments on the 
website. See Todd Welch's diagram of future website plans, Appendix H-3. Task Force 
members include department heads: Sue Seyl, Chris Bostic, John Mead, Marsha Williams, 
Kris White, and Todd Welch. 

The first charge to the committee was should there be a website? This was a non-issue for 
those working on the grant! But the charge forced those in favor of a website to focus their 
arguments and bring the rest of the organization up to speed. 

Later issues were what would the website do, what content should be on it, shifting roles of 
staff as result of grant, and as a place to address other related issues. The group decided to 
focus on polices, procedures, and education of users later. The group started with weekly 
meetings and after the website became available shifted to bi-weekly meetings. 

Initial visit by consultants Beth Sandore and James R. Blackaby. The Inquery search engine 
in use at the Holocaust Museum by Blackaby demonstrated for the first time. There was 
early agreement that the Center for Intelligent Information Retrieval (CIIR), now 
Sovereign Hill Software, Inquery search engine (Available: WWW: 

http://hobart.cs.umass.edu/public.html) was the best way to bring together the disparate 
OHS finding aids and catalogs for web access. Todd Welch began work on Inquery data 
structures and consultants try OHS test databases using Inquery. Both consultants mention 
Photo-CDs as a storage option. 


3/30/96 


Beta test creating slides of various OHS materials sent out to be processed. 


2-3/96 


Flatbed scanner purchased in February and scanning of images, in particular maps, tried 
through March but OHS dissatisfied with quality. The notion of just scanning in the images 
is painfully laid to rest. If OHS could scan items successfully (with high enough quality 
and lowered use of staff time) then there is less need for multiple intermediate collections 
(e.g., slide, Photo CD collections). 


3/96 


OHS begins to consider consequences of making representations of the photograph collection 
available on the website, for example, OHS makes $97,000 annually in photograph 
reproduction sales, can the same revenue be obtained using the web? Can the photographs 
be secured from theft or unauthorized use? Images take up a lot of memory storage and a 
long time to download. Therefore, thought is given to using smaller, lower quality digital 
representations which would reduce mis-use and loss of photo sales. OHS drops hope of 
using digital images to preserve its collections. OHS also drops notion of having a huge 
collection digitally available. OHS begins to consider using the created digital images as 
largely advertisements for the OHS collections. 


4/20/96 


John Mead, Director of Reference and Research Collections, imports serial records and 
begins update of local serial holdings to OCLC, the task includes an issue-by-issue 
inventory. 


5/2/96 


Visit by consultants Beth Sandore and James R. Blackaby. Learn that existing software 
used by OHS for image manipulation is not adequate, need Photo Shop. 
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5/96 


OHS Department of Education committee formed. Generally, meetings are monthly. 


5/96 


The initial draft of the OHS website was developed and demonstrated to the OHS Board 
of Directors. 


6/96 


Beta test of Photo CD, consisting of 33 items from each collection, produced by Lazer Quick 
to discover practicalities involved in digitizing the range of material to be done for the 
grant. 

In the grant proposal OHS originally hoped that CD could be used for both preservation 
(images would be archival quality) and for access via the Internet. With the beta test 
OHS realized that the quality was high but neither magnetic tape (too expensive and 
volatile) or Photo-CD would be good enough for preservation. This decision in turn meant 
requiring the additional step of making archival quality slides as the preservation copy. 
OHS s conclusions were similar to the RLG Commission on Preservation and Access report 
<http://palimpset.stanford.edu/cpa/reports> issued at around the same time. OHS's 
focus shifts to digitalization as a tool to provide greater access to their collections and for 
education, not preservation. 

Beta test forces realization that it takes a long time to prepare a digital representation. 

Beta test forces realization of the need for a change in workflow. Selection and cataloging 
can remain within the individual collection units, but the imaging itself, no matter the 
collection, should all go through the Imaging Technician. 


6/96 


Purchase of Sun Sparc20 from Thurber Technologies 


6/96 


Jaz drive purchased as partial solution to memory storage problem. 


6/96 


Serials librarian hired and entering of local serials holding begins in emest. 


6-7/96 


Initial selection of maps for digitizing begun by Maps Librarian Elizabeth Winroth. 


6/96-2/97 


Full, mock, non-public, OHS WWW page housed temporarily at PORTALS till OHS 
equipment installed. Available: WWW: http://www.portals.org/-chrisb/ (may soon be 
removed 6/97). This web site had 100 images from the beta test mounted as well. Received 
a lot of good feedback including from the PORTALS Executive Director and the Council of 
Librarians. OHS is the last PORTALS member to mount a web site. 


1/96 


Image file naming conventions standardized. 


6/12/96 


Visit by consultants Beth Sandore and James R. Blackaby. 


6/25/96 


Mary Ann Gemegliaro, an assistant to the Director of MIS, appointed Project Navigator. 
She lacks the administrative clout to be effective and is replaced by Barbara Abrams. 



Summer/% Planning underway for OHS staff reorganization to take place in the fall. 
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Summer/% Intensive selection of maps, photographs, and manuscripts begun. 

Summer/% Todd Welch has three interns, uses them to test ideas and work out solutions to problems 

such has how to select manuscript pages to be digitized and determine hours necessary to do 
item level cataloging of digitized images. Todd learned he would need to pare down his 
catalog descriptions if he ever wanted to finish the project. Discussions with the consultant 
Beth Sandore also helped Todd to re-think his ideas on the specificity of cataloging need 
for digitized items. 

7/96 Search for Maps cataloger begun. 

7/96 Museum Artifacts Department begins automation of descriptive information of 5000 items to 

be digitized. A prior grant automated 35,000 records and converted 28,000 black and white 
images to Photo CDs. 

7/96 First slides of manuscript pages created. One consequence was Kris White realized they 

selected too much material given the large amount of staff time and storage requirements 
involved. 

7-12/96 During this period came the realization in the manuscripts area that while digital 

representations were adequate in terms of quality they were not "exciting." Further the 
issues of time to process, storage, and expense begin to penetrate everyone’s thinking. As a 
result, emphasis shifts from a digital representation of the item itself to production of 
digital finding aids. 

8/96 Sun Sparc20 installed by Thurber Technologies, also purchased firewall software and 

installed it at the same time. Thurber provides ongoing system maintenance and consulting 
and designed the OHS web page as well. 

8/96 Todd Welch attends Society of American Archivists, San Diego, CA and an Introduction to 

SGML workshop. Comes back ready to use the Inquery software but the website not yet up. 

8/20/96 OHS beta web site using PORTALS server released Available: WWW: 

http://www.portals.org/ -chrisb/ Todd Welch spends 40% of his time on website 
development from 9/96-11/96 (when OHS annual meeting held) including deciding what 
should be on the page, locating the information or image needed in the correct format and 
preparing it, and hiring an outside company to do the web production. Progress, in Todd 
Welch’s view, was slow. 

9/3/% OHS staff reorganization goes into effect due in part to the impact of the grant on OHS 

practice. Appointment of Barbara Abrams as Project Navigator official (had been Director 
of the Museums Department). All four collections departments (be they library or museum) 
placed under one administrator. 

9/16/96 Map cataloger, Laura Ayling hired. Cataloging on worksheets using MARC standards and 
entering records into OCLC for transfer to the OHS OPAC begins. To give an idea of the 
time involved to catalog a map, the Map cataloger entered 23 records in 1/97, 30 in 2/97. A 
good, full-time, map cataloger, on average might originally catalog 900 maps a year. 
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9/20/96 


Another beta test of Photo CDs produced. Included 19 scanned BxW, color, and manuscript 
maps less than 11x17" in size as well as other collections materials. 


9/96 


OHS judges the existing photographic studio too small to work with large items to be 
digitized. So OHS develops a plan for conversion of Beaver House storage area to a photo 
lab to photograph images, for later digitalization. Innovative work done to prepare maps. 


9/96 


The Coop Map Project begins funded by a PORTALS HEA II-B grant (which phase is 
unclear) with participation by OHS, MCL, PSU, and Reed College. Project plans to add 
1000 partial catalog records (stripped down MARC) of selected maps on a separate Access 
based database searchable by Inquery. Elizabeth Winroth begins search for Maps 
Assistant (hired in November). 



9/23-25/96 Visit by consultants Beth Sandore and James R. Blackaby. 

9/22/96 Method devised for assigning unique ID number to items to be digitized so that they can be 



10-12/96 


tracked and ultimately stored in digital form on a hard disk and the same ID number used 
by a user to obtain a copy of the original (where that is possible). See Appendix H-4. 

Preparation and testing of photography workspace at Beaver House. 


11/96 


Decision made to hire Thurber Technologies to develop the OHS website. Thurber did prior 
work for OHS, was local, new the C code necessary, and OHS Director wanted site to be 
graphically impressive. OHS would need Thurber's help with Inquery installation as 
well. 


11/7/96 


Chief Photographer and Imaging Technician test feasibility and efficiency of Beaver Hall 
studio space for image capture. First attempt to make slides of maps made. 


11/1/96 


Jim Labosier, Maps Assistant hired as part of the Coop Maps Project. 


12/96 


Maps Department completes selection process of maps to be digitized selecting 310 maps 
from a review of 668 collections totally approximately 20,000 maps. Selection criteria 
includes historical significance, geographic coverage, quality of cartography, and 
copyright restrictions. 


12/96 


Photography Department begins conversion of photograph bibliographic records from the 
WANG system to MARC format on OCLC for use on the Horizon OP AC. The Lamb 
Foundation pays for the conversion. Catalogers process and cleanup 300 records of 
individual photographs (Mikki Tint) and 300 records of organized lots (Elizabeth 
Winroth) at a time. 


12/30/96 


New Beaver House storage area photo studio complete. 


1/97 


Photography Department begins automation of the digitized photographs accompanying 
organized lot inventories 


1/97 


Photography Department begins preparation of brief descriptive captions for digitized 
photographs and adds them to cataloged record? 
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1/97 

1/24/97 

1/31/97 

1/97 to 
present 

1/97 

1/97 

1/97 

1/97 

2/97 

Winter 

1997 

2/27/97 

3-4/97 

3-8/97 



Adobe Photo Shop installed and the Image Technician begins manipulation of digitized 
images (gamma and color correction and cropping). 

Chuck McClure, Evaluation Consultant, makes an onsite visit. Requests preparation of an 
evaluation report by OHS. OHS finds the evaluation report writing useful and a timely 
way to assess what they learned and where they are. 

Welch, Todd. (1997, January). Progress Report on Oregon Historical Society portion of the 
U.S. Department of Education, Citizen access to government and other information project. 
(Submitted to PORTALS and Dr. Charles R. McClure, Coordinator of Project Evaluation). 



Major impact from the grant during this period is the development of the website, in 
particular planning how to effectively present OHS information. 

Manuscript Department SGML codes 25 finding aids for testing with the Inquery search 
engine. 

A Nikon slide scanner purchased. 

OHS sends slides of two maps to Lazer Quick to be made into a Photo CD. Test considered a 
failure because result is not detailed enough. 

Cataloging of maps speeds up with the addition of the maps cataloger. At this point, 
worksheets for 111 maps chosen for digitalization are complete. 

Test OHS Web page becomes publicly available: WWW: http://www.ohs.org/ The 
collections departments completed their portions in November but other OHS departments 
were much slower. The Web Task Force agreed that everyone must be ready before the page 
was publicly released. 



Conversion by Ameritech of WANG system bibliographic records of the photography 
collection to MARC completed. Editing of the collection-level and item-level 
bibliographic records is on-going. 

Next attempt at creating slide images from maps. Slides of 40 maps made during January 
and February. OHS discovers need to resolve uneven lighting, to insure the full coverage of 
the map, and to resolve proper exposure issues. Started using two cameras to cover the 
entire space of the map without moving the map (changing lighting and alignment). The 
magnetic wall solution for photographing maps developed. 

Richard Yost, while converting Photo CD images to hard disk storage, exhausts OHS 
storage capacity. A multi-gigabyte drive is added. 

Production of manuscript finding aids aided by four practicum students acquired via a course 
taught at Portland State University in Archival Administration. 



Ryan/McClure Onsite Evaluation Report: Citizen Access to Government and Other Information August 12 , 1997 

3/16-19/97 Museums on the Web conference in Los Angeles attended by Sue Seyl, Barbara Abrams, 
Marsha Williams and Todd Welch. 

3/26/97 Construction of OHS website's Kids page begins, not yet publicly available. 

Spring 1997 Todd Welch creates a template in Microsoft Access for photographic image descriptions. 

Fields include: image file name, CD code and image identifier (a standardized naming 



4/97 


convention allows path access to file on hard disk) , descriptive title (including date of 
photograph), and organized lot number. 

Selection of photographs is complete. 


5/97 


Museum Artifact Department begins reading artifact images on Photo CD into hard disk 
memory, by 6/15/97 have 600 images processed. Process not without pitfalls, process takes 
a day or two per CD, one CD had to be read and cropped three times. 


4/21/97 


Dwight Patterson, MIS Senior Engineer, is hired. 


5/97 


Johnyne Wascavage, Serials Assistant hired, will assist with addition of local serial 
holdings project (funded under next phase of the grant). 


6/97 


SGML viewer, Panorama Pro from SoftQuad purchased (a free viewer is also available as a 
plug in). For further information see: http:/ / www.softquad.com/products/panorama/pan- 
free.htm 


6/10/97 


Television station KGW, channel 8, does a short feature on the OHS grant project and web 
page. 


6/97 


An uninterruptable power supply purchased for Sun Sparc20 as well. 


7/97 


Contract signed to use Inquery search engine. 


7/31/97 


Argus database contains 47,604 artifact descriptions compared with slightly over 20,000 
artifact descriptions at the beginning of the grant period. 


8/97 


Ameritech loads WebPac software making it possible to access the OHS Horizon catalog 
from the website. 



Staff Development Activities 

This sub-section summarizes efforts by grant participants to obtain education and training for 
staff to better implement the grant objectives. OHS was the most active of the grant participants in the 
pursuit of staff development activities, a sample of their activities includes: 



Date 


Activity 


11/1/95 


Charles McClure, Evaluation Consultant, meets grant principals in Portland. 


12/19/95 


Charles McClure, Evaluation Consultant conducts conference call with grant principals. 
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1/96 All staff computer conference at OHS 

3/5/% OHS staff conference on automated collection management in the Madison room, the focus is 
on the HEA II-B grant 3-8 PM 

3/14/96 Initial visit by consultants Beth Sandore and James R. Blackaby 
5/2/% Visit by consultants Beth Sandore and James R. Blackaby 

6/12/96 Visit by consultants Beth Sandore and James R. Blackaby 

Summer 

1996 Todd Welch attends "Automating Finding Aids" workshop of the Society of American 

Archivists in Tempe, AZ. 

8/96 Todd Welch attends Society of American Archivists, San Diego, CA attends two day SGML 

workshop which included construction of a DTD. 

9/23-25/96 Visit by consultants Beth Sandore and James R. Blackaby 

10/96 Todd Welch learns about Encoded Archival Description (EAD) tag library and further 
information about the Berkeley Finding Aids Project. For further information see Daniel 
Pitti's (1993 to present) progress reports, Hensen (1995), Wilson (1995 to present), and the 
finding aids themselves (University of California, Berkeley, 1993 to present). Other 
finding aid projects are underway including Duke University, Harvard University, the 
Library of Congress, and Yale University. For further information see the references 
section. 

1/24/97 Chuck McClure, Evaluation Consultant, makes an onsite visit. 

3/16-19/9 7 Museums on the Web conference in Los Angeles attended by Marsha Williams, Sue Seyl, 
Barbara Abrams, and Todd Welch 

8/97 Todd Welch will attend a EAD workshop. 

The retained consultants' and the PORTALS Evaluation Consultant spent significant portion of their 
OHS contact time in staff education and training. OHS staff made extensive use of the Internet and 
listservs on professional topics (EAD, LSCH, Archives and Archivists lists) to obtain needed 
information and training. WWW pages frequently mentioned include the Commission on Preservation 
and Access <http://palimpset.stanford.edu/cpa/reports>. Library of Congress, American Memory page 
<http://lcweb2.loc.gov/amhome.html>, Northern Arizona University's Special Collections 
Department <http://www.nau.edu/~cline/speccoll/imagedb.html> and the United States Holocaust 
Memorial Museum in Washington, DC <http://www.ushmm.org/>. In the maps area, Columbia 
University (see for example, Oversized color images, <http://www.columbia.edu/dlc/nysmb/>) and 
the British Columbia Archives and Records Service (BCARS) 

<http://www.bcars.gs.gov.bc.ca/cartogr/general/maps.htm>. Others at OHS relied on colleagues at 
former places of employment or schooling. OHS staff previewed books at Powell's Technical books. For 
additional staff development activities see list compiled by one staff member (Richard Yost) of his 
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activities, see Appendix H-5. For additional comments see the ’’Continuous Staff Development Needed 
at All Levels” section under ’’Lessons Learned.” 

Project Work Products 

This sub-section lists and describes the major products, services, and other outcomes as a result of 
the grant. 

Five OHS collections were part of the digitalization process: the artifact collection from the 
OHS museum and the manuscript, photographs, maps, and oral history collections from the OHS 
library. Appendix H-6 ’ Collection Pages for Web Sites” depicts the near term list of collections 
involved. The OHS work products can best be understood in terms of the series of steps necessary to 
digitize each of these collections. Appendix H-7 ’ DOE Project Work Flow” graphically depicts the 
process. These steps were essentially the same for each of the collections: 

• Select the material to be digitized. Selection processes varied among the units involved. 
The Museum Artifact department intended to digitize all of its collection so the issue was 
which to do first. The other OHS collections units intended to select a representative 
portion of their collections and the issue was what criteria to use in selection. For further 
discussion of selection criteria see Lessons Learned, Selection of what to digitize is complex, 
below. 

• Make slide images of the items selected. Initially there was hope that photographs, 
manuscripts, maps and other flat items could be quickly scanned into digital form. OHS 
experiments determined that the quality of the scanned image was not satisfactory. 

Further there was questions about the archival value of other formats such as Photo CD 
and computer hard disk. After much trial-and-error in July 1996, OHS adopted 35 mm 
slides (using T-64 slide film) as the preferred archival medium, at least for now. Photo 
Craft (320 SW Stark Street, Portland 97204, (503) 225-0515) developed the slides. OHS 
made one slide for each item except in the case of a map where OHS made five slides (1 full 
size and one of each quadrant of the map) for each map. See Appendix H-8 ’’Map Inventory 
records” for a partial illustration of the recordkeeping involved. 

• Convert slide images to Photo CDs. OHS views Photo CDs as an acceptable offline backup 
storage medium in case of erasure of items stored on a hard disk. Other options were optical 
storage (in original grant proposal but not purchased) or another hard disk (expensive) or 
magnetic tape backup (known to be unreliable with digital images because tape loses image 
data over a short time). Slides could be used as a backup (particularly with the recent 
OHS purchase of a slide scanner that yields high quality digital images). However, in the 
event of a need to recover lost images (e.g., due to a broken hard disk) retrieval of the 
images from the slides may be too cumbersome, time consuming, (and from a preservation 
perspective the less handling of the slides the better). 

OHS could choose to directly access an image stored on a Photo CD (typically a group of 
CDs on a jukebox) from the Internet. Direct access of the Photo CDs is somewhat slower 
(this is changing), would require purchase of a jukebox, and require technical knowledge 
regarding networking of the jukebox which OHS does not presently have. However the 
principal reason for not using direct access from the Internet to the Photo CDs is image 
quality. The Photo CDs completed so far by OHS still need further editing (cropping to 
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remove among other things a border, color correction, gamma correction, and other 
adjustments) before use. It is not clear to the evaluators why slides couldn't be scanned, 
edited and then stored on Photo CD ready for WWW access. For further information on 
Photo CDs see Kodak in references. 

The present process has the image technician, after examining the slides for quality errors, 
prepare the slides for Lazer Quik. He checks the Photo CD images upon return from Lazer 
Quik and stores the returned slides. Note the labor involved. Lazer Quick (1134 SW 5th 
Street Portland, OR 97204) converts the slides to Photo CD. The process takes on average a 
week costing $79 per CD of 100 images ($.69 per image and $10 for blank CD). Each Photo 
CD stores a high resolution surrogate image of the original. An algorithm is then applied 
to the image which allows the Imaging Technician to view the digital image from 5 
different resolutions : file sizes: 128 x 192 : 72K, 256 x 384 : 288K, 512 x 768 : 1.13MB, 1024 x 
1536 : 4.5MB, and 2048 x 3072 : 18MB. The Imaging Technician uses the third size (512 x 768 : 
1.13MB) when processing the image from the PhotoCD. See Appendix H-9 "Image capture 
to Photo CD, Photo CD to WWW'for further detail on the slide and photo CD process. 

The image technician, recommends OHS move this process in house if planning extensive 
and ongoing Photo CD production. OHS would need to purchase CD-ROM writer, blank 
disks, and software. 

• Enhance image quality. Digitized images need cropping (typically to remove the border 
surrounding the digitized item) and may need other quality improvements like gamma 
correction and color correction despite careful preparation of items when shooting slides . 

At present, OHS performs this editing on the digitized image (in this case using the Photo 
CD image uploaded to hard disk) using such software as Photo Shop. 

• Convert the Photo CD images to hard disk storage. Hard disk storage permits the fastest 
present access to images for use on the WWW (which is still rather slow). Conversion steps 
include image cropping (using Photo Shop and a Kodak viewer), gamma correction, color 
correction, saving the file in processed archival ( average resulting size is 1.13MB in TIFF or 
PCD, decided not to use the base times 4 image), thumbnail (average of 30K file size, GIF) 
and full-size (average of 100-200K file size, JPEG) formats and managing the file sizes, and 
digital watermarking. At present, this process takes from 3-6 minutes per image. The 
Image Technician is working on a way to batch process the full-size and thumbnail images 
after editing the archival images which should significantly reduce the staff time 
involved (the batch process can be run overnight). See Appendix H-9 "Image capture to 
Photo CD, Photo CD to WWW"for further detail on the slide and photo CD process. See 
Appendix H-10 "Web Scanning /Imaging Register" for a partial illustration of the record 
keeping involved. 

• Catalog, to some degree, the items selected and add to digital databases. OHS cataloged 
each of the items selected to some degree, generally in a brief format adapted from some 
national standard. An existing partial catalog of the photograph collection is being 
converted to the MARC standard as part of the grant. The maps librarian cataloged maps 
to be digitized to MARC standards. In a separate PORTALS HEA II B grant (subsequent to 
the present grant) OHS participates in the Coop Map Project. 1000 maps will receive 
abbreviated MARC cataloging and be stored on an Access database. The museum 
department catalogs its artifacts using brief records stored on an Argus database. The 
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library catalogues its monograph and serial collections using MARC standards and is 
adding local serial holdings to the OCLC database and the OHS Horizon catalog as part of 
the grant. But in general, OHS rejected full MARC cataloging as too labor and time 
intensive or in certain cases the existing OHS cataloging format was in use prior to the 
development of the MARC standard. For an example of the cataloging involved with maps 
see Appendix H-8 "Map Inventory Records." The decision about the level of cataloging in 
turn affects where OHS digitally stores the resulting catalog descriptions. Simply, if not 
up to MARC standard it will not be on the OHS catalog. As a result, a variety of paper 
based and now digital databases exist. Major databases include the Argus database of 
artifact descriptions, and several Microsoft Access databases under development. 

• Prepare finding aids to relevant collections. The items selected for digitalization from the 
library (manuscripts, photographs, maps) were representative of much larger collections. 
Finding aids for the individual items and these larger collections also needed preparation. 
Initially, this aspect of the project was secondary but recently digital finding aids assumed 
prime importance because they give a better sense of the collections available to the 
external user and are searchable. 

• Captioning: Link finding aids, catalogs, images (where possible) and the collection via a 
search engine. The Inquery software promises to enable the search for information in all of 
the various databases produced inhouse of finding aids and catalogs, these in turn will 
provide access to existing digitized images (for the remote user) and the collections 
themselves. The Inquery search engine will permit searching across the various OHS 
collections for the first time. See Appendix H-ll "Collection Search Engine" for a mock-up 
of the screen a user might see when using the Inquery search engine. The Inquery software is 
about to be purchased after much delay by OHS. An "Image Catalog Database," see 
Appendix H-12, will link bibliographic description, collection location, caption, and other 
relevant information (e.g., permissions to use) for presentation on the website. 

• Create a website to provide access to digital surrogates of OHS collections. Work products 
presented on the website will include: the digital images, the OHS Horizon and other 
catalogs, finding aids, and the Inquery search engine. In additions, the website will contain 
information on OHS, its departments, services, and policies. Appendix H-3 graphically 
depicts the full scope of OHS website plans. Todd Welch did the initial website design. 

But OHS contracted with Thurber Technologies for production design and maintenance. 

OHS staff wishing to add to or alter the website must seek permissions using a form (see 
"Web Site Proposal for New Material/Appendix H-13). 

Kris White estimates the costs of digitizing a manuscript page at roughly $30 including slide creation, 
Photo CD mastering, cropping and editing, conversion to hard disk storage, and preparation of 
descriptive captions. This figure includes staff time in processing but not selecting an item or website 
maintenance. This figure assumes the infrastructure is in place (computers, staff, staff training, 
software, workflow, etc.) OHS will make a more precise cost estimate with further experience. 

Table 1 "Summary of OHS Work Products” presents the grant targets, and current status of the 
work products arranged by department. Consult the OHS website < http://www.ohs.org/> for the 
latest information. The current status portion of Table 1 makes use of benchmark data regularly kept by 
John Mead, Director of Reference and Research Collections, for the project (for examples see Appendix 
H-14, Sample Benchmark Statistics Sheets). 
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Table 1 



Summary of OHS Work Products 



Department 


Grant Targets 


Current Status 


Museum 

Artifacts 


5,000 of 85,000+ artifacts 
made accessible via 
digitized images and 
cataloging of some type. 


Select: The museum intends to have its entire 
collection on Photo CD and ultimately on the 
website. Selection is complete. 

Convert to slides: Prior to the grant the museum 




created 28,000 slides. 2,000 additional slides made 
during the grant period. 

Convert to Photo CD: Prior to the grant 28,000 slides 






converted to Photo CDs. During the grant period 
2,000 slides converted. 

Convert to hard disk: The Image Technician cropped 






gamma and color corrected 600 items and placed them 
on a hard disk ready for WWW display. 

Catalog: 20,000 items entered into Argus database 
prior to grant. Now a total of 47,604 items cataloged. 
Finding aids: Exist for some of the items and planning 
underway to make them accessible on the website. 


Maps 


300 images of maps 
appropriately cataloged 
drawn from a collection of 
30,000 maps. {Note: No 
commitment made to convert 
maps to GIS format.) 


Select: 334 maps selected from 557 collections of 
approximately 20,0000 maps by examining 668 maps. 
Selection completed by 12/96. For a tentative listing 
see: <http://www.ohs.org/maps/mapaid.htm>. 
Convert to slides: OHS scanned 19 maps (11x17 or 
smaller) and converted 40 maps to test slides. Making 
slides from the rest of selected maps is a next step. 
Convert to Photo CD: This is the next step. OHS used 
19 scanned maps in Photo CD beta test. 

Convert to hard disk: Next step in the project. 
Catalog: Over 183 worksheets cataloged to MARC 
standards prepared, with 155 entered into OCLC and 
the OHS catalog, Expect to complete by grants end. 
Finding aids: Some finding aids exist and there are 
plans to make them accessible on the WWW. Coop 
Map Project plans to add 1000 brief catalog records on 
a separate Access database searchable by Inquery. 
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Manuscripts 


2,500 pages from 450 
collections made accessible 
via digital images and 
cataloging of some type. 
Overall Manuscript 
collection is 14,000 cubic 
feet. 


Select: Reviewed 19,530 document pages from 477 
collections, selecting 3,422 items for digitalization, 
later de-selected 375 pages, for a total of 3,047.. 
Convert to slides: OHS converted 1.391 manuscript 
pages to slides and reviewed 1,200 of them. 

Convert to Photo CD: 553 manuscript pages prepared, 
106 of them enhanced. 

Convert to hard disk: 10 sample items are on the 
website see: <http://www.ohs.org/mssample.htm>. 
this is the next step. 

Catalog: 154 collections cataloged, 720 images 
cataloged (of 3,047 items). 

Finding aids: 36+ collection inventories prepared and 
coded in SGML for use with the Inquery search engine 
and the WWW. More in preparation with the 
assistance of 4 interns. 16 finding aids are on the 
website see: <http://www.ohs.org/mssample.htm> 


Oral 

Histories & 

Sound 

Collections 


100 oral histories including 
40 full biographies of state 
legislators and several 
others averaging 16 hours in 
length and related 
materials. {Note: No 
commitment made to 
transcribe these histories 
into text.} 

Total collection of 500+ 
items. 


Select: Selected a core group of central figures first 
(these were obvious), then tried to show the variety 
of types of government officials, then those that 
would impress the listener. Selection is complete. 
Convert to hard disk: Audio quotes from each 
history, an image of the person will eventually be 
digitized. 

Cataloe: MARC records for spWtpd histnrips addH 
to catalog -- not complete. 

Finding aids: Prepared a list of all people mentioned 
in the entire oral history collection. Prepared 
outlines and proper name indexes for each (average 
20-30 pps.). These aids need to be entered into MS 
Word. Intend to make these aids accessible on the 
WWW via Inquery search engine. 
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Photographs 


2,500 photographs from 358 
lots totaling 150,000+ made 
accessible via digital 
images and cataloging of 
some type. There are 
upwards of 2.5 million 
images in this collection 
with on average 200 new 
photographs being donated 
a week. 


Select: Completed selection of 2162 images in 4/97 bv 
reviewing 358 collection inventories and then 
examining 102 collections containing 63,266 
photographs. 

Convert to slides: 2619 slides made (some were re- 
done) 

Convert to Photo CD: 25 Photo CDs made containing 
2162 images made 

Convert to hard disk: 485 photos imaged, 200 
enhanced. 

Catalog: Conversion of the photographv catalog on 
the Wang to MARC records and made available on 
the OHS Horizon catalog begun in 12/96. A template 
using Microsoft Access for photographic image 
descriptions created and data entry begun in 1/97. 
Fields include: image file name, CD code and image 
identifier (a standardized naming convention allows 
path access to file on hard disk) , descriptive title 
(including date of photograph), and organized lot 
number. 

Finding aids: 300 photo inventories (finding aids} 
exist, 200 of which are in WordPerfect. 


All 


Develop the website. 

Make digital images of the 
collections, the catalogs, 
and finding aids accessible 
on the website. 


In 2/97 OHS website available: WWW: 
http://www.ohs.org/ 

Inquery search engine to link various catalogs and 
finding aids into a meta-index is about to be 
purchased. OHS already developed data structures 
and formats already. 


Serials 


Add 1313 local serial 
holdings to OCLC and 
Oregon Union List of Serials 


602 done, found OHS did not own 25 as of 6/1/97. 
Intends to update the State Library's ORULS 
database as well. 


Oregon State 
Archives 


Create WWW series Echoes 
of Oregon 


There are twenty three items in the Echoes of Oregon 
History set as of May 15, 1997 
<http://arcweb.sos.or.gov/echoes/default 
echoes.html> including: Willamette Cattle Company 
Agreement, 1837; Account Book, 1840; Certificate for 
Boarding a Lunatic, 1845; Sandwich Islander Tax Bill, 1845; 
Shark Broadside, 1846; Declaration of Intention, 1849; 
Animal Bounty Bill, 1849; Resolution to Expel Catholic 
Missionaries, 1849; Description of Land Claim, 1849; 
Defendant’s Request, Whitman Massacre Trial, 1851; 
Willamette University Trustees' Report, 1853; An Act to 
Prevent Sabbath Breaking, 1854; Petition to Allow the 
Thomas Family to Stay in Oregon, 1854; Prohibition Petition, 
1854; Committee on Education Report, ca. 1854; Indian Agent 
Regulations, 1855 Arms Inventory, 1855; Letter about Oregon 
Volunteers, 1856; Church of the Brethren Petition, 1856; 
Abstract of Votes, 1857; Request to Open Indian Lands, 

1857; Half-Breed Citizenship Bill, 1857; Memorial to 
Congress, 1858; and Divorce Petition, 1858. 
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OHS learned the current limits of the digitalization process as the result of the grant. One of 
the early casualties was the hope that digitalization would contribute to preserving the collection 
(the technology is not yet ready). But a principal plus, meeting a key grant goal, was improvements in 
citizen access to OHS materials. Improvements in access available to PORTALS members and citizens 
as a result of HEA II-B grant include: 

• A sense of the range and diversity of formats of OHS’s collections via image 
representations, textual descriptions, finding aids, catalogs, and the Inquery search engine. 

• A better sense of whether or not OHS is an appropriate place to come to research a specific 
topic due to expanded cataloging of the collections, expanded development of finding aids, 
the addition of a website, the Inquery search engine, and e-mail contact. 

• A way for a remote researcher to prepare a research plan in advance for a trip to OHS due 
to e-mail, website, and finding aids. 

• A more complete picture of OHS's products and services prior to physical arrival on site due 
to the information presented on the website. 

• The ability to search the collections, and find linkages across them, in ways not previously 
possible using the Inquery search engine. 

Overall, OHS should be commended for their efforts to meet grant goals and target objectives. 
Dissemination of Project Results 

This sub-section identifies specific ways the grant participant disseminated project findings to 
the local, state, and national communities and the profession including publications, videos, press 
releases, presentations, external training sessions, etc. 

OHS is the most active of the grant participants in disseminating information about their grant 
efforts. These efforts ranged from teaching courses (Archives Management and Public History) at 
Portland State University (which OHS gained four practicum students this year who produced finding 
aids for the project); systematically registering the website with a range of search engines; hosting 
visits by area museums, educational institutions, and high tech firms (these "show-and-tell” visits 
occurred as often as once a week in the fall 1996 and Spring 1997 period by one estimate); presentations 
at regional professional meetings (Todd Welch, Kris White, and Sue Seyl presented separate papers at 
the Spring Northwest Archivists meeting May 14-17, 1997 in Spokane, WA) and national (Sue Seyl 
will do a panel at the fall Society of American Archivists); a brief feature on a local television station 
(KGW, channel 8); and press releases (See Appendix H-15 ’’Oregon’s Rich Cultural Heritage On-Line” 
for an example). A brief article, ”OHS spins a web at www.ohs.org” appeared in Oregon History 
Magazine , see Appendix H-16. Kris White notes the beginning this spring 1997 of phone calls and other 
contacts inquiring about various aspects of the HEA II-B grant project. For a sample of individual staff 
dissemination activities, see the lists compiled by Kris White and Richard Yost in Appendix H-5. 

Sue Seyl also made a point of noting the need for dissemination within OHS not just to the rest 
of the world. The OHS web task force played an important role first in focusing attention on the need 
for building awareness within the organization and then as a vehicle for disseminating the latest 
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project news. Another important mechanism was the all staff computer conference held in January 1996. 
The Onsite Evaluator did not have time to interview those OHS members not participating in the 
project directly to get a sense of their awareness and knowledge of grant activities. 

Evaluation Efforts 

This section identifies specific ways OHS used to evaluate the project. OHS is the most active 
of the grant participants in devising means of self-evaluation for the HEA II-B grant project. Examples 
include: 

• Since December 1995 John Mead collected benchmark data bi-weekly (tied to the OHS pay 
period) on portions of collections digitized. 

• OHS monitored staff hours spent on the project. 

• OHS developed a number of work forms and written procedures to digitize archival 
materials? 

• Project Navigator, Barbara Abrams, appointed in September 1996 to coordinate grant 
activities across OHS departments and improve overall communication. She immediately 
created a set of H what if” spreadsheets to track:, the original proposal, changes to it, and 
what if scenario impacts on the budget. This enabled her to see better the evolution of some 
of the grant pieces, particularly coming on in mid-stream. 

• OHS formed a Web Task Force (met weekly then bi-weekly) and a Department of 
Education Team to assess where OHS is in the process, communicate across units, air 
disagreements, and adjust accordingly. 

• The Web Task Force creates a form (see Appendix H-13) which codifies the policy and 
procedures mechanism for adding new information to the website. 

• Todd Welch did a series of flowcharts on the digitizing portion (see Appendix H-7) and 
WWW design portion of the grant (see Appendix H-3) that were useful visualization tools. 

• Argus database produced reports on progress on creation of artifact text descriptions. 
Horizon catalog produced reports on status of various cataloging efforts. 

• Log analysis software to assess website usage tried but not found satisfactory (See Appendix 
H-18). 

• Appropriate use of external consultant evaluations by Beth Sandore, Coordinator for 
Imaging Projects, Associate Professor of Library Administration, University of Illinois and 
James R. Blackaby, Visiting faculty. University of Victoria and President of J.R. Blackaby. 
The consultants filed reports on their 3/14, 5/2„ 6/12, and 9/23-25/96 visits with OHS and 
later received by PORTALS and the Evaluation Coordinator. 

• Welch, Todd. (1997, January 31). Progress Report on Oregon Historical Society portion of 
the U.S. Department of Education, Citizen access to government and other information 
project. (Submitted to PORTALS and Evaluation Consultant). 
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OHS used the production of the report as an opportunity to take stock of its experience to 
date. 

OHS’s concern with evaluation was evident to both of the evaluators during their respective visits. 

The fundamental flaw with OHS’s evaluation efforts to date is that they have not involved 
their users. OHS’s evaluation efforts have been focused on understanding or improving its internal 
operations or use of information technologies. OHS does not appear to have asked its users for input, 
assistance, advice, or help aside from the obvious efforts of the development department. For example, 

• OHS did not ask for input from its users regarding selection of items to be digitized. User 
based evaluation could have been combined with a game, "See your favorite work from the 
OHS on the Internet." OHS could have raised money, "Have your favorite OHS work on 
the Internet for a donation of $100!" 

• OHS did not explain onsite or on the Internet its efforts to digitize its collections asking for 
advice, explaining constraints as they were discovered, preparing users to appreciate the 
results, educating users as staff became educated, inviting users to educate OHS staff. 

• OHS did not consider how it might interact with its new users on the Internet or promote 
onsite interaction via the Internet. OHS does not list staff names, phone numbers, office 
addresses, and e-mail addresses on its website. 

Lessons Learned 

The OHS project might be characterized as a series of challenges, experiments, and 
adjustments. Principal challenges included: 

• Digitize Selected Items from OHS Collections: Discovering, often by trial-and-error, the 
processes, procedures, and technical options available to digitize the various OHS 
collections. Then fixing on an efficient and effective digitalization method to meet grant 
objectives. Then exploring possible uses for the resulting digital images given the now 
known constraints of quality, time, resources, and staff needed. Specific lessons learned and 
discussed below include: the selection of collection items to be digitized, deciding how to 
digitize items selected, and, given that hard-won knowledge, identifying uses for the 
digitized images. 

• Digitize and expand finding aids to OHS collections: Adjusting OHS’s course when 
organizational learning suggested that finding aids would prove as valuable, if not more 
valuable, to users as digital images. Discussed below are specific lessons learned including 
why the Inquery meta-index matters. 

• Website Development, Establish a virtual OHS on the Internet : Establish an OHS Internet 
presence by creating a website. Populate the website with OHS products, services, policies 
and procedures. Explore the meaning of a virtual OHS and its consequences for the 
organization. This portion of the project is just getting underway so the lessons learned are 
few. 
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• Learn, as an organization, from digitalization: Establishing and maintaining a learning 
organization, continuously training staff, and adjusting structure, policy, and procedure 
where necessary. Specific lessons learned discussed below include: the need for continuous 
staff development at all organizational levels, that systemic change is not frictionless, but 
archive, museum, and library cultures can co-operate, and the need for organizations to 
learn how to best use relationships with outsourcers 

This sub-section identifies significant lessons learned by OHS from each of these principal challenges 
which may also be of interest to similar organizations in other settings. Two challenges not directly 
addressed yet by OHS relate to OHS users both physically present and virtual. What will be the 
consequences of OHS's digital presence (on its website) for OHS staff and its products and services 
offered onsite? Second, who are OHS's digital users (on the website), what OHS products and services 
do they want, what can these digital users offer OHS? Facing these challenges are clearly on OHS’s 
horizon, they are lessons about to be learned and are considered in the Recommendations sub-section 
rather than here. 

Digitize Representatives of the Collections 

Digitizing representative items from the artifact, manuscript, map, oral history, and 
photograph collections was the first challenge OHS faced in the minds of the participants. 
Discovering, often by trial-and-error, the processes, procedures, and technical options available to 
digitize the various OHS collections. Then fixing on an efficient and effective digitalization method 
to meet grant objectives. Then exploring possible uses for the resulting digital images given the now 
known constraints of quality, time, resources, and staff needed. Specific lessons learned include: the 
selection of collection items to be digitized, deciding how to digitize items selected, and, given that 
hard-won knowledge, identifying uses for the digitized images. 

Selection of collection items to be digitized 

In terms of what to digitize, the Onsite Evaluator heard a number of different selection criteria 
being used including: 

• Visually interesting: Does the image convey information that a user could not obtain any 
other way, or obtain as rapidly any other way (this includes manuscript text). 

• Copyright free (or permission to image granted). In the case of maps, this meant produced 
by the government or published prior to 1920. 

• Contributes to an overall view of Oregon history with attention to significant highlights, 

• Ease of digitalization (flat, 11x17 or less, item would photograph adequately), 

• Fragility of item to be digitized. If too fragile not touched. If item would be otherwise 
inaccessible and it could be handled with care by OHS staff, then the item was a candidate 
for digitalization. 

• Ease of cataloging. Images that were digitized were cataloged. If the item was already 
cataloged or part of a collection in which some prior organization was present it was more 
likely to be considered for digitalization. 
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• Link in an obvious way, to other OHS collections likely to be digitized or with planned 
exhibits. For example, a planned exhibit on the Oregon trail might link a map of the 
Oregon trail to a diary entry of an early settler, to an artifact and book on the topic. 

• Illustrates the collections diversity in terms of format or content, 

• Frequent past use of materials by OHS collections staff and onsite users, 

• Federal requirements (e.g.. Native American artifact holdings needed to be identified.), 

• Grant focus on government information. 

The selection criteria for items to be digitized, cataloged, and made accessible on the website were 
largely collection driven. This is not surprising given the collections departments role in the grant’s 
development and implementation. 

OHS could have chosen other selection criteria than those driven by the collection. For 
example, 

• Will the web image generate revenue or help the development unit to target a certain group 
when fund raising? 

• Did website users request the image, were selection criteria collaboratively developed by 
website users and OHS? Already OHS has had to develop new collection and donation 
policies in response to website public demand. See "New Deed of Gift” Appendix H-17. 
Already e-mail requests for manuscript information are greater in number than letter 
requests (but still less than phone requests which the highest). 

• Does the teacher (or the OHS education unit) think the image best conveys the educational 
point trying to be made? 

• Did a market survey suggest users or OHS members wanted images of a certain type? 

• Does the image represent an historically underserved or neglected group? 

• Is the image in some way like ones used by OHS’s peers on their web sites? 

These and other selection criteria may well become important in the future as other stakeholders 
become active and the website and collections become more digitally accessible. Will OHS, and in 
particular the collections departments (and those responsible for digitizing images), embrace these new 
selection criteria and these new selectors, particularly when external to OHS? 

Deciding how to digitize items selected 

A complex, evolving, approach 

OHS’s method for digitizing selected items from its collections evolved throughout the grant 
period. Several factors influenced OHS’s evolving approach including: 
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• OHS is in some ways a typical historical society with significant portions of its collections 
not organized. Those portions which are organized are the result of isolated, 
uncoordinated, small, grants which unintentionally promote a tendency toward 
idiosyncratic, customized, localized, information processes (i.e. cataloging and workflow) 
and specialized staff roles (encouraging a perverse form of job security). The best historical 
societies, OHS among them, resist these tendencies but they are a presence that all 
instituting change must recognize. 

• Changes in the information technologies necessary to digitize the collections. In general, 
organizations seeking to digitize their collections today will find improved quality, 
reduced costs, more options present more clearly explained, and cheaper storage. 

• Better appreciation of what digitizing involves and the value of various alternative 
options gained by OHS as a result of trial-and-error learning. 

• Better appreciation for how digitized images could and could not be used by OHS as a result 
of trial-and-error learning. 

The evolution towards a best method of digitizing for OHS is by no means over. Neither the technology 
or OHS's learning curve has peaked yet. 



Approaches considered 

Approaches to digitalization that OHS that OHS considered, with some brief comments about 
each, include: 

• Scanning (not artifacts of course) — quality was not high enough for use or archival 
purposes, would require extensive handling of collection. 

• Digital camera -- quality was not high enough for use or archival purposes, fast processing, 
reduced handling, relatively inexpensive, would re-consider when quality improves and 
then cost comes down. 

• 4x5 slide — high quality, good archival preservation, high cost, not by itself a total 
solution (not digitized), not the way industry went so companion pieces (e.g., slide scanner) 
may be more costly. 

• 35 mm slide — acceptable quality and archival preservation, lower cost, industry standard 
approach, not a complete solution still need to digitize -- OHS currently using this as one 
element of its approach. 

• Photo CD — assuming photo (slide) is high quality CD image is high quality, quick and 
easy to create, low startup costs, reasonable production costs, uses existing equipment in 
part, can outsource where equipment not owned, can output to printer, digital form, need for 
post-processing due to apparently incomplete control of cropping color, etc., creates yet 
another intermediate surrogate (still have to digitize to control cropping and color), input 
is a surrogate (the slide) rather than the object. — OHS currently using this as one element 
of its approach. 
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• Slide scanner -- converts slide directly onto digitized hard disk storage. OHS recently 
purchased a Nikon scanner. The Image Technician says results are high quality. This 
raises several potential changes in procedure. The Photo CDs cannot be used for Internet 
access at present principally because the images need editing (cropped and color corrected). 
Could the slides be digitized onto the hard disk, edited and then be made into Photo CD 
images suitable for direct access on the Internet? This would alleviate some of OHS's 
storage problems. Alternatively, is the process of making a hard disk copy of the slide 
image easy enough that a Photo CD image copy is no longer needed as back up? These 
questions are under consideration by OHS at the moment. 

At present, for each item to be digitized OHS creates six surrogate representations. First is a slide 
photograph, then a Photo CD image, and finally thumbnail, full-sized, and archival digital images 
stored on a hard disk. That is, OHS creates six new surrogate collections (each to contain 10,000 
representations) in order to digitally represent the 10,000 items chosen from its collections for the grant. 

Current process 

Each item to be digitized must be selected (requiring a professionals time). Each item must be 
found and retrieved from its present location or the slide copying equipment brought to the item. Each 
item is assigned a unique ID (within the file name length restrictions of the computer operating 
system). This ID number is used throughout the digitizing process and will become part of the file name 
of the image when stored on the computer hard disk. The Image Technician batches items selected and 
retrieved by size (making it easier to photograph). Often the item had to be temporarily stored 
(meaning finding the space for temporary storage and meeting each item's storage requirements) before 
being photographed. An image logbook to link object, ID number and position/number of the roll of film 
needed to be devised along with a place to comment on problems when shooting the slide. The item once 
photographed must be returned to its original location. A means for keeping location information with 
the item needed to be devised. When the Image Technician received the slides they needed to be 
identified, checked for quality (chemical residue, need for cropping, etc.), checked to see if all sent out 
returned. Then the Image Technician organizes the slides into lots of 100 to be put onto the Photo CD. 
OHS stores the slides separately from the original and the image on CD) also requiring staff time, 
storage, and documentation. When the CD-ROMS return the Image Technician checks to see if the 
images are all there and of acceptable quality. These steps (among others) required more staff time in 
planning and implementing that was originally imagined. 

Documentation: A full time job 

Keeping track of the item and key events (what was sent, when, to who, received back when, 
etc.) and work products (slides, cd-rom tracking numbers and disk location, file directory on a hard disk) 
in the process of digitizing an item from OHS's collection was a necessary and important but cumbersome 
task. No doubt as more organizations become involved in digitizing the record keeping process will be 
automated. Until then, organizations will need to allocate staff time to simply keep track of the 
current status of the materials to be digitized (and their location after digitalization). 

Large storage requirements 

OHS could only leam by experience about the memory requirements and types of storage (CD, 
slide, hard disk) necessary to digitize a collection. The grant target was to digitize 10,000 items. OHS 
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stores a digital representation of an item in three formats processed archival, requiring on average 1.13 
MB in a TIFF or PCD format; a full size, requiring on average 200-300K in JPEG format, and thumbnail, 
requiring on average 20K. Rounded up storage for one item is 1.5 MB. To digitize 10,000 will require 15 
Gigabytes of storage just for the images. This does not include software to prepare or use the images. 

This does not include software such as Inquery and its large inverted files and other software needs. 

At present, OHS is deciding how to manage the immense storage requirements necessary to meet 
proposed grant objectives. To meet the targets set by OHS for the grant will require a lot of memory, to 
digitize more of OHS’s various collections will require a great deal of memory. "To continue to buy 
gigabyte hard drives is not an option." remarks OHS MIS Director. It may be that OHS will have to 
wait for memory technology to catch up with OHS's need. 

Current criteria when considering a change of approach 

A current summary of criteria OHS employs when considering a new approach to digitizing its 
collection would include the following (not ranked): 

• Affordable ~ there is a threshold above which OHS simply cannot afford to be interested. 

• Incorporates existing OHS technology (reduces cost, adds value to present equipment, 
reduces training). 

• Range of output quality — high quality for archival purposes, future needs, and to avoid 
having to re-digitize; lower quality so that (a) collection is not appropriated, for example 
if photographs on the WWW are too good a quality users will not pay for copies of the 
originals a current significant source of revenue; (b) some feel that users should, for their 
own benefit, have to come to OHS to use its collections (there is no substitute for physical 
contact with the collection, there are other items of potential interest to users not available 
on the WWW which the user could find out about if physically present at OHS). 

• Improved processing ~ Does the approach reduce the time it takes to digitize an item or 
make it available to users, improve workflow, is it simpler or easier to use (requiring 
limited training), does it reduce handling of materials, or reduce need for recordkeeping? 

• Compact storage and transmission — Does the approach reduce the amount of digital 
storage needed and (as a result) increase the speed with which an image can be dent to a 
user? 

• Archival value — Does the approach improve preservation of item or its surrogate? 

• Meets standards and avoids need to re-invent the wheel, now or in the future — Does the 
approach require re-doing already digitized items, will the approach require a re-do in 
the future? 

The search for an ideal approach is by no means over at OHS. But the expectation that such an ideal 
will ever be met is tempered by experience. 
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Identifying uses for the digitized images 

Originally there was the hope that eventually, starting with this grant, a substantial portion 
of the OHS collection and finding aids would be digitized at high enough quality for OHS and its users 
at a distance to rapidly access and use OHS materials in both old and new ways. One way of viewing 
the history of the grant however, is as a series of reductions in expectation, but with the possibility 
that the day after tomorrow all original hopes might still be fulfilled (due to some new technological 
improvement). An early reduction in expectation came when OHS concluded that at present digital 
images could not be used for preservation purposes due to a combination of cost, processing time, storage 
requirements, inadequate quality, and continued questions about the stability of the storage media, 

The prevailing view at OHS today, after the trial-and-error of learning how to digitize its 
collections, is that digitizing images takes too much time, staff, and resources; that high quality 
images take up too much storage (1.13MB was minimally acceptable for use by OHS and remote 
scholars) and take too long to view over the web (they take a long time to download), and their use once 
downloaded could not be controlled by OHS. Why would anyone physically come to the institution 
with its collection digitized? Why would anyone pay for a copy of an OHS photograph when they 
could download it free from the website (an important source of revenue for OHS)? How could we 
prevent the use of OHS material without attribution? How could we prevent the use of OHS material 
without alteration? With these considerations in mind, why digitize? 

The prevailing view at OHS today is that the principal use of digital images (or video or film 
clips) is as advertisement, maybe for education (yet to be explored), but not for scholarship, information 
management, or preservation. The exception to this view is the Museum Artifacts department which 
began digitizing its collection first years back, intends to digitize the complete collection, and finds the 
digital images essential for locating objects in the collection and for insurance purposes. At present, 

OHS has two reasons for digitizing images, one fairly firm, the other still to be tested. There is general 
agreement that digital images on the OHS WWW site make attractive, eye-catching, "with-it,” 
statements. The use of digital images conveys the impression that OHS is not some dusty old museum 
but a place you ought to visit in the 21st century. The use of digital images on the OHS website attract 
new and old users to come and visit the collections in person. Digital images can also be used to give 
potential visitors a sense of the depth and variety of OHS's collections both in content and in form. A 
second reason to use digital images is in k-12 education. This potential use is about to be tested with the 
introduction of the Kids Page on the OHS website. Images, not necessarily of the highest quality, catch 
and hold children’s attention and can best convey key ideas. 

Digitize and Expand Finding Aids to the Collections 

OHS gained practical experience with digitizing their collections in 1996. They concluded that 
for the present (who knows what technology will bring) that the production of digital finding aids 
were more valuable to their users that digital representations of the collections themselves. The cost, 
storage requirements, and processing time was high, often only yielding minimally adequate quality 
digital representations. The University of California, Berkeley reached a similar conclusion earlier 
and began the Berkeley Finding Aid Project. See Daniel Pitti’s (1993 to present) progress reports, 
Hensen (1995), Wilson (1995 to present), and the finding aids themselves (University of California, 
Berkeley, 1993 to present). Other finding aid projects are underway including Duke University, 

Harvard University, the Library of Congress, and Yale University for further information see the 
References section. 
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With the decision to focus on digital finding aids OHS next faced the catalogers conundrum: 
faced with a hugely impossible job (organizing all of OHS's collections) do you catalog a few things 
well (i.e., use MARC and other national or international standards), or catalog more adequately (i.e., 
the Coop Cataloging Project which uses a partial approximation of the MARC standard), or make 
inventories, lists, even public service desk cheat sheets (with varying degree of local control) to survive 
the public's insatiable need for information? This conundrum is made more challenging by different, 
conflicting, and even non existent standards for non book/serial materials (i.e., artifacts). Beth 
Sandore, the OHS consultant, was of great help to OHS staff as the decided what to do using her 
experience to indicate when "well” was necessary and "adequate" cataloging good enough. What has 
overshadowed these decisions, and what makes the digital environment distinctly better, is the 
promise of meta-indexing software such as Inquery coupled with the WWW. 

Why the Inquery Meta-Index Matters? 

OHS is made up of a set of different collections including photographs, manuscripts, books and 
serials, maps, oral histories, artifacts, and various finding aids to all these collections. Over the 
years, OHS developed various partially completed databases using different software on different 
computers including the Ameritech/ Horizon library catalog containing the libraries book, serial, and 
other major collections; the Argus database of artifacts, various Microsoft Access databases (including 
Map groups, photography inventories (finding aids — to be done), captions and link information for 
digital images to be accessed by WWW, the manuscript item catalog, manuscript finding aids (to be 
done) and even in WordPerfect like the oral histories outlines and proper name indexes). In addition, 
there are various OHS paper indexes that could be converted to databases including the vertical file 
index, biographical index, pioneer index, and genealogical index. 

WWW browsers such as Netscape or Microsoft Explorer provide a common window through 
which to view data. The SGML or HTML tags tell these browsers how to interpret or view various 
types of data in different formats. So here is OHS with lots of data, including some that it has an HEA 
II-B grant to digitize (read, make capable of being viewed by a web browser). Further OHS has all 
these databases which help locate OHS’s collections in different formats. Needed: Some software 
that will look to the user like it is searching one big database but in reality the software will really go 
out and search all the databases (on all the different formats and computers OHS has) that index 
OHS's collections and report its findings back to the user. 

The software that goes out and searches all of the different digital databases OHS has and 
then reports its findings back to the user’s web browser is the Inquery software. Without this software, 
all of the existing OHS databases and the collections they index remain isolated from each other. 
Without this software, databases developed to different standards can not be jointly searched for 
useful information. With this software, staff across all of OHS’s collections and OHS users will have 
new and unprecedented access to the collections. Topics can be searched across the different types of 
collections present at OHS. 

With the improvement this software promises there is a strong incentive at OHS to produce 
additional finding aids because their use and accessibility in no longer limited to OHS staff, multiple 
standards are now less important, and even "survival’’ cataloging can be used. 
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Establish a virtual OHS on the Internet 

What are the consequences of having so much of OHS accessible by users not just staff? 

OHS's web presence is only several months old and the institution is only beginning to consider 
and respond to the consequences and impacts of virtual OHS. Already there is a certain amount of 
concern, and a sense of venturing into unknown territory as more and more of the OHS backrooms becomes 
available on the WWW page. How will user communities respond? What will it mean for OHS, for 
staff, for users? The answers to these and other questions represent lessons about to be learned. Worth 
mentioning briefly are three catalysts which contributed to the establishment of OHS's website: a 
champion, a task force, and a collaborative outsourcer. 

Todd Welch got tired of waiting for an OHS web page. So one weekend, at home, on his own 
computer, while he was building a page for the Northwest Archivists (where he is 
Secretary /Treasurer) he designed a prototype OHS web page. The impact at OHS was immediate, 
suddenly staff had something concrete to help them understand what this WWW stuff was. ..and it was 
created by one of their own. 

The second catalyst was the creation of the Web Task Force by the OHS Director. Significant 
was the Director's first charge to the group: Should OHS have a website? For people like Todd Welch, 
the website was overdue. But for others in the organization websites were quite new. The task force 
provided a forum and a vehicle for those in the know to educate others within OHS to a website's 
importance and significance to OHS. The task force also provided a place to work out problems and 
disagreements. 

The final catalyst was the decision to outsource website production to Thurber Technologies. 
Thurber was new to the business and ready to please. OHS trusted Thurber due to prior work Thurber 
had done for OHS. OHS’s technical staff were over extended with grant and other activities. It was 
important at this developmental point that no one was seen to "own” the website within OHS. At 
present, with the website launched and initially established, there is active discussion about bringing 
web site development back in house. In any case, Thurber Technologies played an important role in the 
development of the OHS web presence. 

Learn, as an Organization, from Digitalization 

Perhaps the key challenge OHS faced was the creation (or maintenance) of an experimental 
environment and a learning organization. An environment that tolerated mistakes but with a 
mechanism to learn from them and to recognize success and integrate it into organizational practice. A 
flexible staff with a "loose" hierarchy to enable the most effective use of talent and interest. An 
organization capable of tackling the new, or new to the organization, without falling apart and at the 
same time, maintaining the valued old processes and services. The Onsite Evaluator speculates that a 
hidden asset was OHS's existing organizational structure. A structure common to many contemporary 
archives and museums. A core staff funded from the operating budget provides stability and core 
services while a critical mass of short term experts, funded by soft money, experiment. Some of today’s 
experiments become tomorrow’s core services. So while there is considerable respect for the old, there is 
thoughtful appreciation of the new. OHS managed quite well, there is a flexibility and spirit that 
even a short term visitor senses. 
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Continuous staff development needed at all levels 

OHS learned that a major project like the grant will require education and training for all 
staff: those implementing the grant as well as those affected by it, those implementing a new 
technology as well as those managing the technologists, the "technically challenged” as well as the 
"technically dangerous," and the volunteer staff as well as the paid. Training needs are continuous and 
some needs may be unexpected. Contracting with consultants to devote part of their time to staff 
training can be very useful if thoughtfully planned. OHS does not have a training officer, unit, or fixed 
operating budget. OHS does seem to have an environment in which dumb questions are welcome and 
thoughtfully answered, at all levels of the organization. 

Need for Technology Training: Basic Computer Literacy and Advanced Technical Skills 

There is the usual range of technical competence within OHS. At the extremes, notes MIS 
manager, there are the 10% who are "technically challenged" and the 10% who are "technically 
dangerous.” The "technically dangerous" are those who need to know, want to know, or think they 
know more than presently do about some technical aspect. The MIS Director recently noticed a definite 
healthy increase in the "technically dangerous” as a result of the grant. Even the MIS staff would like 
to take classes on advanced UNIX topics offered by the local Sun education unit. The "technically 
challenged” lack basic computer and network literacy skills and may even resist basic training. 

Barbara Abrams, Project Navigator, notes that this project "raised the bar" across the staff in terms of 
minimum expectations of what staff should be able to do. Even volunteer staff must adjust when OHS 
shifts to a digital environment. At present, OHS recognizes the need for volunteer training and for the 
creation of technological jobs suitable for volunteers. But because the digitizing process in still new and 
not yet fixed, planning for volunteer recruitment, role re-definition, and training in this area is on hold. 

Need for Management Training 

The focus of staff training for projects of this type is often on technological training but 
manager’s need training as well. Barbara Abrams, Project Navigator notes that, "We assumed that 
good managers before would be could managers in a new technological environment and in many ways we 
shot ourselves in the foot.” Senior managers, and even the Board, all could have used training in simply 
managing a project of this size including: basic project management skills, project management software 
and Excel spreadsheets; how to staff at appropriate levels; budgeting; and how to document a project of 
this type. Managers need technical skills as well. Managers needed to understand what their 
subordinates did, converse with them, identify staff technophobia, recognize the difference between 
staff technophoria and why the staff needed a particular piece of technology (hard or soft) now! 

Several managers want detailed training in core software in use (e.g., Access and Inquery) particularly 
when the software becomes a standard in the ongoing work of staff unexpectedly (e.g., At the beginning 
of the grant, the importance of Access and Inquery to the project was unknown.). These managers 
cautioned that the need to know technology would never come at the right time and would always seem 
costly in terms of time, energy, and sometimes money. 

Use of Consultants Time Partially for Staff Training: A Plus if Thoughtfully Done 

OHS experience suggests that one of the roles a consultant can and should play in the early 
phase of a project like this is as a staff trainer and educator. If thoughtfully done, this portion of the 
consultant’s time can be as beneficial as specific advice about certain problematic decisions or the 
consultant’s outside evaluation of grant progress or next steps. 
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A Department of Education requirement, resulting from the grant contract negotiations, was 
that OHS hire a consultant. OHS eventually hired two consultants, Beth Sandore and James R. 
Blackaby, who became a team as a result of working with OHS. How OHS hired the consultants, the 
consultants themselves, and OHS's approach to using the consultants served as an intensive staff 
training exercise. OHS identified a set of questions that OHS project staff had at that point in the 
project. OHS then addressed those questions to the consultant candidates. When OHS combined Beth 
Sandore and James R. Blackaby’s answers, most of the OHS staff's had useful answers. The process of 
identifying OHS staff questions, engaged everyone, made OHS aware of the need for staff 
development, and specifically identified the type of training necessary. 

When the consultants arrived onsite they appear to have immediately seen the need for an 
educational component and generously offered training and direction. The consultants readily shared 
their mistakes (OHS chose both consultants because they had hands on experience) as well as their 
successes (e.g., Jim Blakaby narration of the development of the Inquery search engine at the Holocaust 
Museum.). This more than anything else helped OHS staff to relax, not panic, to realize that they 
were not going to make "one crucial mistake," and not be "so uptight when making every decision." 

OHS arranged for the consultants to meet widely within the organization, and with groups as well as 
individuals, spreading further what the consultants knew. The two consultants had different 
backgrounds and ways of talking about a problem. This often enabled them to bridge the many cultures 
involved in the project including lay, technical, library, and museum cultures. Overall, most felt the 
OHS experience with the consultants was (to quote Barbara Abrams), "one of the highlights of the first 
year of the project." 



Staff training pays many hidden dividends. Barbara Abrams, Project Navigator, points out that there 
is a hidden cost in the way the grant is staffed and trained in that current decisions promote staff 
specialization. Specialists demand and command higher salaries, specialized positions are harder 
and take longer to fill, or can't be filled and projects fail. Staff that are cross trained can fill in for 
gaps in personnel at a reduced cost. 



How and when money is allocated internally for what remains an issue. Conflicts arise when 
someone at the height of his learning curve needs a resource but must interact with others whose 
attention must necessarily be episodic or has not yet had time to catch up to the needed resource's 
importance. 

The notion of appointing a project navigator, a sort of analogue to the product champion in 
business, on its face, seems like a good idea. Project navigators might well work for many grants. But 
then issue arises, who to appoint? Clearly the person must have both authority and influence, they 
must have time to devote to the project navigation task, and they must be technically competent. For a 
smaller project, someone possessing all of these characteristics might be found. The issues here are two: 
the project affects every aspect of what OHS does (one must have authority, influence over all, and 
time for all) and second the technical concerns differ enough among the formats being digitized that a 
difficult job of keeping up verges on impossible. Wanted: Person with the authority and influence of the 
Director (to cover the problem of systemic change), plenty of time (after all, what do directors do 
anyway), with technical mastery of digitalization of multiple formats. Issue summarized: in the 



Hidden Costs of Over Specialization 



Systemic change is not frictionless 
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absence of superior knowledge, authority, and time all have had to and will continue to have to 
contend, learn, and live together — not to mention learn to forgive. It is not clear whether the OHS 
project participants realize yet that an ideal project navigator will not be found (at least not until 
enough historical societies and libraries go through what OHS is trying to do now). 

There were persistent complaints about delays in the purchase of software or hardware (some 
of which was budgeted in the original grant proposal) necessary to complete some phase of the project. 
The delays were evident to the onsite evaluator, but their cause was not. Were the delays and resulting 
conflict normal for an organization experimenting with new technology? Is there mis-communication 
between requestor, authorize^ and purchaser? The evaluators suggest that the situation be 
investigated and an accounting offered to OHS staff. 

Multiple lines of authority in a fluid work environment creates an ideal state for lack of 
accountability (or the appearances of such among co-workers). Managers expressed concern to the onsite 
evaluator that they did not always know whether certain work or requests for resources for such work 
was authorized or whom to ask. Many expressed the need for a manager who could cajole, prod, or force 
where needed the completion of targeted deadlines. One successful model in similar environments is a 
form of management by objectives approach. Each staff member develops a set of objectives to be met 
say quarterly and at the end of the quarter explains their success for failure in meeting the objectives. 
Those in charge sign off on the objective setting and later evaluate whether the employee met the 
personal objectives set. Important objectives and targets can be changed at any time with approval of 
all those to whom the staff member reports (assuring a continued, realistic balance). This approach 
better defines chain of command, allows all to see the workload for each staffer, sets up an audit and 
accountability trail for each person, yet preserves individual initiative and responsiveness to shifting 
work requirements. 

Perhaps at the stock taking session recommended below solutions will emerge because OHS, as 
an organization, is experienced enough to reapportion authority, time, resources, and technical mastery. 
Or perhaps experience will enable all to recognize that the above described frictions are the norm for 
21st century organizations facing the unknown and learning together. 

Archive, museum, and library cultures can co-operate 

The grant brought the museum and library worlds together as together they worked out 
problems associated with digitalization. Each culture was encouraged /forced to learn enough about the 
other to effectively work together. The museum and library found common interests and began bridge 
building. A key case in point is preparation of the catalog and finding aids. The cataloger trained to do 
monographs suddenly must learn how to catalog manuscripts, serials, photographs, maps, oral 
histories, even one day, artifacts. Along with each of these formats is an associated tradition, culture, 
and personalities -- all needing to be mastered. The payoffs to all are obvious. In particular, with the 
advent of the Inquery meta-index, OHS and its users will be able for the first time to find and use 
materials from all of OHS's collections. The desirability and possibility of collaborative projects now 
is easier or in some cases possible for the first time. 

Digitalization seems to enable or force the interaction of a range of previously isolated cultures. 
This is not limited to cultures based on race or geography, but cultures such as the archival, library, 
museum and computer cultures. One OHS staffer remarked that, "digitizing information for the WWW 
is bringing institutions and departments together that had very distinct operating practices in the 
past." The need for distinct ways of operating is breaking down in the face of a process of presenting 
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information from each of these operating units on the WWW that does not differ much among these 
previously separate units. Organizations and their staff must recognize that the new culture that 
emerges from the meeting of two older cultures may be quite different than either. Are the days of 
specialized manuscript, monograph, map, and artifact catalog units over? Do separate museum, 
archive, and library units still make sense? Organizations must prepare their staff members for these 
culture contacts so that the most can be made of them. 

Learn how to best use your relationships with outsourcers 

At key points during the project, external vendors, consultants, and outsourcers played 
significant positive roles. Deciding what should be done internally within the organization, finding 
external partners, communicating clearly what is needed, and cultivating ongoing relationships are 
important, learned, skills. OHS learned how to make use of external partners successfully. In the case 
of vendors, leveling the playing field between vendors who did prior work for OHS and new bidders by 
offering thorough briefings on OHS needs and then giving serious consideration to all bids mattered. 
Thoroughly checking out vendors including visits to local offices, checking with the Better Business 
Bureau was useful. Regular communication with vendors to find out what is new and share new OHS 
activities also helped. 

At key points during the project, external vendors, consultants, and outsourcers played 
significant negative role. OHS appears to have above-average working relationships with its vendors 
which it works hard to maintain. But OHS has also dealt with vendors who did not meet their own 
deadlines, would not set a deadline, and vendors that OHS needed more than the vendor needed OHS. 
These situations created uncertainty and extensive delays. Others planning to use the same technology, 
or planning to use current information technologies in general need to know that uncertainty, delay, and 
dependence are the norm. Legal action is sometimes limited but often the only recourse available. 
Barbara Abrams, Project Navigator, notes that at the start of the grant period attorneys were seldom 
involved in small and medium sized contracts. Today attorneys routinely read all OHS contracts. 

Lesson for OHS: It is Time to Combine its Technical and User-based Approaches 

In sum, to address the question what should be digitized and why, OHS has gained extensive 
experience with the question of how to digitize but no experience yet with what its digital patrons 
would like in digital form. In library parlance, OHS has adopted a technical rather than public 
service approach to exploring these questions so far. The limits of the technical approach by itself are 
well known: the explorer learns something of what is possible and nothing of what is needed. The 
lesson for OHS is that it is time engage its virtual patrons in a discussion to discover what is needed so 
that OHS and its outsource partners can use its hard-won knowledge of what is possible to make it so. 



Next Steps 

This sub-section suggests next steps to be taken as this grant period comes to an end and to 
advance the project's objectives after the funding period. OHS staff identified a number of other next 
steps during the interviews with the Onsite Evaluator including: 
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• Digital Images: Meet grant targets for digitalization of representative portions of its 
collection: Despite present hard disk memory concerns, OHS is committed to meeting the 
targets it set for digitalization of representative portions of its collection by the end of this 
grant period. In order to meet grant targets to create digital images for 10,000 items, OHS 
must address several problems. Principal problems include: find an acceptable way to store 
processed archival images; finish conversion of Photo CD images to processed archival, 
full-size, and thumbnail images (Onsite Evaluator told existing hard disk memory is 
adequate to store the full size and thumbnail images used for the website and required to 
meet grant targets); protection of hard disk stored images from erasure or corruption (which 
has been an problem for the artifact images placed on the hard disk so far); complete 
conversion of oversized maps to slides; and advance the oral histories portion of the grant. 

• Finding Aids: Use Inquery search engine to link existing digital catalogs and finding aids: 
Purchase Inquery and then use the software index and provide access to existing digital 
catalogs and finding aids. This will be exciting for remote and onsite users, it will also be 
exciting for staff, for example, planning exhibits. 

• Finding Aids: Increase production of digital finding aids: The development of digital 
finding aids will expand once the use of Inquery as a meta-index to digital finding aids is 
successful. 

• Finding Aids: Continued processing of less organized collections: A collection has to be 
organized or otherwise split into manageable units in order to receive further processing (for 
example, be examined for items to be digitized). There are "gems" contained within these 
less organized collections that all the staff, over time, would like to make accessible. 

• Website Development: Move website development in house, get other OHS units on board, 
improve web access, and consider the websites strategic role: Todd Welch demonstrated a 
prototype web page which propelled OHS to develop its own site by outsourcing the work 
to Thurber Technologies. OHS is at a crossroads: do they continue the Thurber relationship 
or find the funds to hire a web master in house. In any case, other units within OHS will 
become a part of the website efforts. The OHS website needs to be made accessible to the 
majority of Oregonians who use dialup technology that can not handle the present 
graphically intensive OHS website. OHS needs to consider the role of the website in its 
strategic planning. To do that OHS needs to consider who it users are, what their needs are, 
what they can contribute to OHS? 

• Address the impacts of the rapid organizational changes brought about in part by the 
grant: The current need to institutionally embed lessons learned about effective ways of 
coping with ongoing change caused by the grant or information technologies regularly 
surfaced during the onsite interviews. Diagnostics include policy documents being prepared 
then policies waiting for adoption by senior management, the perceived need to regularize 
information technology training, discussions about the job description of the Project 
Navigator, delays in resolving the image storage question, and the ongoing discussions 
about standardizing the digitizing workflow across formats (and departments). 
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• IT and OHS a new equation? OHS needs to consider the importance of information 

technologies to a modem historical society and take actions to reflect that importance in 
organizational structure, staffing, and resource allocation. The grant increased awareness of 
information technologies, their resource requirements, their required staff roles, and their 
potential importance to OHS. The next step is to consolidate this new awareness into 
embedded organizational practice to maximize information technologies utility. 

These areas are likely to be pursued in the coming months and as this phase of the grant ends. OHS 
also received another installment of funding from the DoED HEA II-B grant. 

Recommendations 

OHS has made substantial progress towards its grant objectives. OHS has moved as an 
institution from the 19th to the 21st century in the process. The evaluators offer the following 
recommendations based on the onsite visit and subsequent discussions and in the context of a successful 
grant effort. The fundamental challenge as this portion of the grant ends is how to consolidate the 
learning resulting from the grant in institution wide structures, policies, practice, and services so as to 
continue OHS’s blend of solid service and innovative experimentation. 

Complete Grant Targets for Digitizing Selected Items from OHS Collections 

In order to meet grant targets to create digital images for 10,000 items, OHS must address 
several problems. Principal problems include: find an acceptable way to store processed archival 
images; finish conversion of Photo CD images to processed archival, full-size, and thumbnail images 
(Onsite Evaluator told existing hard disk memory is adequate to store the full size and thumbnail 
images used for the website and required to meet grant targets); protection of hard disk stored images 
from erasure or corruption (which has been an problem for the artifact images placed on the hard disk 
so far); complete conversion of oversized maps to slides; and advance the oral histories portion of the 
grant. The grant targets are a worthy goal and clearly a prod to staff to advancing the project. It is 
time for the final sprint after a long race! 

OHS is concerned that the digital representation of its photography collection (as well as 
other collections) will be used without OHS agreement. Digimark offers a digital watermark which 
may offer some protection. Digimark is a local company (down the street from OHS) with a national 
reputation. Contacts made by Richard Yost should be pursued. A donation from Digimark would be 
nice. A co-partner in future problem solving in this area would be much more desirable. 

The evaluators sense that OHS feels it has pushed present affordable technology in this area 
to its limit. OHS must now await further technological developments. The technological requirements 
are clearer now than at the start of the project. First, a digital camera with high enough resolution to 
efficiently capture preservation quality images of a range of items from oversize maps to manuscript 
pages and photographs, to three-dimensional artifacts. The goal is to digitally capture an item, no 
matter the format, at high quality and without the need of duplicate collections (one collection of 
originals, one because of quality, one because of cost, one because of rapid retrieval, one because of 
preservation, etc.). Existing means of storage and editing tools seem adequate for post-processing and 
editing of captured images. Second, the digital camera should be linked to a cheap, permanent (secure 
from image deletion and stable enough to serve as a preservation medium), storage mechanism with (at 
the same time) rapid retrieval capability (of various quality/resolution/sized images) for fast Internet 
accessibility. The existing web interface seems to be adequate to presenting OHS’s digital information. 
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Third, the digital camera and adequate storage must be connected via high speed information transfer 
to OHS's patrons desktop (or living room). OHS and patrons must await the availability of an 
affordable high speed network to take advantage of the digital representations of OHS’s collections. 

Digitize and expand finding aids to OHS collections. Link with Inquery Software 

The Inquery meta-indexing software promises to make both complete (e.g., OHS Horizon 
catalog of MARC records) and partial (e.g., a newly created digital finding aid to an unorganized lot of 
photographs) collection organizing aids accessible to onsite and remote OHS users. This is most useful 
to historical societies where their collections are not likely to ever be fully organized, but always in 
the process of becoming so. Further, these various collection organizing aids can be digitally combined 
to allow searching for information across collections. This meta-indexing feature may permit the 
discovery of exciting connections across collections not previously known. 

To get there, to achieve the promise suggested by the meta-index's use, will require a great deal 
of work. Priorities, in a step-by-step fashion, from this distance seem to be to: 

• Train more people in the requirements, capabilities, and use of the Inquery software. 

• Give key personnel a conceptual understanding of the contemporary database and 
information retrieval concepts which underlie Inquery (and HTML/SGML). 

• Consider the need for uniform finding aid standards across the collection departments to 
maximize finding aid utility in a digital environment. 

• Devise rational principles for selecting the degree to which a given collection (or item) 
will be organized — why MARC treatment, why a standardized finding aid, why no 
organization at all (yet)? 

Note, we did not mention the purchase of Inquery, the development of prototype applications, and the 
expansion of the production of digital finding aids (both from the conversion of existing non-digital 
aids and the creation of new digital finding aids). These needs seem clear enough at OHS not to require 
the evaluators attention. 

What concerns the evaluators is the danger of isolating the knowledge necessary to effectively 
use the software in a person or unit within OHS. How OHS collections are organized and accessed is a 
core function of the organization. This requires that a range of staff within OHS be able to understand 
enough about this new method of digital information organization to converse, to imagine, to 
coordinate, and to use this emerging tool together. Not only does the staff need to know more than they 
do about this emerging way of digitally organizing data, so do OHS's patrons. In sum, it is not too soon 
to start transferring out from the technical personnel within OHS's collections departments the 
knowledge that the rest of OHS and its public will need to be innovative. It is not to soon to consider 
what are the organizational structures, policies, standards, and procedures necessary to maximize 
OHS's use of this new capacity. 
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Website Development: Establish a Virtual OHS on the Internet 

OHS is laboring to give birth to a digital representation of its physical self: must it be the 
same, should it be different, should it always be tied to mama, who should have a say? To treat 
website production as a technical exercise to create an advertisement for one’s (physical) self is surely 
to miss the point. The evaluator’s suggest OHS give greater imaginative attention to who/what its 
digital progeny will be? To do so consider the following suggestions: 

• Move website development back inhouse: The evaluators believe that the need to rapidly 
and conveniently alter the content of the web page(s) will eventually make retaining the 
web page and its production within OHS the most attractive option. When OHS begins 
producing web based programs for OHS users the need for ready inhouse access will be 
almost mandatory. Producing and managing the website inhouse will require a webmaster. 
In addition to funding the position, problematic areas to address include the web masters job 
description, reporting relationship, relationship with the MIS department, etc. The 
solutions offered by a good inhouse webmaster who is well positioned within the 
organization will be worth the effort to obtain one. 

• Develop and enable inhouse capacity to alter the OHS website: There are two suggestions 
implied here, one technical, the other managerial. First, to develop the skills (such as 
ability to code in HTML and SGML) among key staff to make simple alterations to the 
website. Second, make the organizational changes necessary allow and enable appropriate 
staff to manage their unit s portion of the OHS web site. 

At present, with the web production done off site by Thurber Technologies, the incentive to 
learn how is removed or reduced. The evaluators recommend department heads designate 
staff to be allowed to alter web page content and that these staff receive training as 
necessary in OHS concerns for quality and standards as well as how to code properly in 
HTML/SGML. Drop the permission forms, they are too cumbersome. The webmaster 
recommended above can help here by identifying key staff within each unit to manage the 
unit s portion of the website, obtain and coordinate necessary training, establish standards 
for each unit to follow in such areas as common "look and feel,’’ and monitor for compliance. 

• Improve website accessibility: The Onsite Evaluator accesses the OHS website from home 
using a 28.8 modem and local telephone lines. This method of access is equivalent to the 
way most of Oregon’s present Internet users from home, libraries, schools, even offices access 
OHS's website. For these users, access to the OHS site is painfully slow due to the high use 
of graphic images when it works at all (7 out of 10 times the screen froze when the Onsite 
Evaluator used the site). At minimum, the evaluators suggest that OHS should create a 
text only option on its homepage giving OHS users access to the website’s textual content. 

Further, the evaluators suggest that OHS ask for feedback on its site from visually 
handicapped users. Comments from these users could be useful in designing a better site for 
those with normal vision as well. Making the OHS website compliant with Americans 
with Disabilities Act standards increases access and improves design. For further 
information see: International Committee on Accessible Document Design (ICADD). 
Available: WWW: http://www.ucla.edu/ICADD/html2icadd-form.html or U.S. General 
Services Administration. Center on Information Technology Accommodation (CITA). Public 
and private resources section. Available: WWW: http://www.gsa.gov/coca/pub_res.htm 
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• Create an OHS Intranet: Many organizations are finding (1) that the web is a useful 
supplement to other forms of internal organizational communication, for example, internal 
policy development (2) that this communication is best done on a separate, institution only, 
web site. If the present web site is on the Internet, the proposed internal site is the Intranet. 
The evaluators believe an internal web site might be useful to OHS — if for no other reason 
than it is a great place for staff to practice their HTML /SGML! 

• Focus on OHS’s digital users and the digital environment: OHS should seek to incorporate 
mechanisms on its website to engage its digital users in a dialogue about what digital OHS 
should be, the types of information and services that should be offered, etc. OHS should 
adjust its internal practice to effectively respond to feedback from its digital users. The 
MIS director will quickly become tired of filtering and channeling digital users feedback 
(as is presently the case) if it grows as it should. 

Perhaps the Web Site Task Force, after each member has created their own web page (the exercise is 
enlightening), should pause and consider: must the Virtual OHS be the same as the physical version, 
should it be different, should it always be tied to mama, who should have a say? 

Information Technologies: Peripheral or Core Functions of a Modem Historical Society? 

The MIS and Communications Department consists of two people, one recently hired (4/21/97). 
This unit handles selection, purchase, installation, troubleshooting, and maintenance of computer 
hardware (including 75-100 personal computers) and software for all of OHS. The department’s major 
responsibilities include providing computing resources for the grant, accounting, membership, 
development, the point-of-sale and inventory systems for the store, e-mail for staff, and management of 
the Novell network. In addition, the department is responsible for OHS’s telecommunications needs 
including telephone, fax, and pagers. Web site development and maintenance of the Sun Sparc20 are 
outsourced to Thurber Technologies. Much of OHS’s technology base is funded by grants and other one- 
time funds. OHS staff’s use of information technologies increased dramatically over the past three 
years. 



Has the selection, installation, use, and replacement of information technology (IT) and the 
training of staff and patrons in IT’s effective use become a core OHS function? If so, is the information 
technology function authorized, staffed, and funded on par with other core OHS functions (and 
consistent with current and future IT needs)? Has OHS taken an inventory of the staff’s information 
technology skills, has OHS matched need to know with current level of IT knowledge? Has OHS begun 
a program of staff training correct mis-matches? The evaluators pose these questions without certain 
knowledge of the answers. But the rapid introduction of information technologies to OHS, their 
continued high cost, their apparent importance, and the intended present and future use of these 
technologies suggests the need for strategic attention to IT’s new place in the organization. 

Focus on OHS's Digital Present and Future Users 

Now that OHS is to the stage where there is "something to show" its users, the evaluators 
suggest it is past time to shift OHS’s attention to its users both those that are virtual as well as those 
onsite. The evaluators suggest OHS pursue several strategies: 
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• Explore the meaning of a Virtual OHS with your users: The basic questions to be asked 
about the digital environment are not much different from those asked in the non-digital 
world: What do OHS patrons need, what resources does OHS have of use, and how can 
OHS patrons help OHS to thrive? OHS needs to create onsite and on net forums to ask, 
listen, and discuss with users the meaning of what a Virtual OHS ought to be? 

• Experiment and prototype particularly with targeted groups already on the Internet. OHS 
did not begin to grasp the importance of a website until Todd Welch created a prototype 
web page. Your patrons are no different, stimulate their imagination. 

• Prepare the institution to address user response. It is pointless to ask and not be prepared to 
respond, to learn, and to share the dialogue of question, response, and lesson with others. 
This implies an organizational commitment to ask, to efficiently and effectively respond, 
to evaluate and adjust, and to educate others including staff, patrons, and the professions. 

Ask, listen, learn, act it sounds simple but it is not always easy to do. The good news that all those who 
would succeed at establishing a virtual presence on the Internet are engaging in these simple processual 
steps. Invariably they discover the messy, baffling, uncertainty associated with the beginning of any 
creative act. The bad news, particularly for those who have built before asking, is that the babel from 
the building may drown out the voice of need and innovation. The expenditure of staff energies and 
resources in the building (n this case of a digitally based OHS) translates too readily into commitment 
to the existing edifice however imperfect and unwanted. 

Learn, as an organization, from digitalization: Stop and take stock 

In the fall, perhaps after, or as part of the final report for this first HEA II-B grant, the 
evaluators recommend the institution as a whole pause, and take stock of where it was and where it is 
now, what it learned along the way, and next steps that need to occur to encourage continued success. To 
an interested outsider, OHS learned a great deal as it moved from material to digital culture, or at 
least added digital culture to its storehouse. The danger is that the learning will not be consolidated 
and embedded into institutional practice. 

Get the Word Out 

Perhaps one of the byproducts of moving rapidly, as Sue Seyl notes, "from the 19th to the 21st 
century" may be that OHS does not know how much it has to share with fellow historical societies, 
museums, archives, and libraries. Many businesses would have an interest in the OHS experience as 
well. The Onsite Evaluator noticed the tendency of OHS staff to compare their work with the current 
"big names." The comparisons miss the point when it comes to disseminating what OHS knows. The 
"giants" of the field may not need your findings (although the evaluators would disagree). But the 
organizations just like you, of which there are many, believe they are alone in their interest, alone in 
their problems, and without recourse because no one of their size, or type, or situation is disseminating 
what they know. Get the word out! 

All of the above recommendations are in the context of a very successful organizational effort to 
digitize images from key portions of the OHS collections, develop and link via meta-indexing a range 
of finding aids to the collections, establish a presence on the Internet, and modernize a range of 
institutional practices. The evaluators commend the Oregon Historical Society for their efforts which 
will dramatically increase access to Oregon’s rich heritage to all of Oregon and beyond. 
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VI. Overall Project Conclusions and Recommendations 



Conclusions 

This report documents substantially improved citizen access to federal, state, county, and city 
government information in Oregon as the result of the U.S. Department of Education HEA II-B grant to 
the Portland Area Library System (PORTALS). This grant enabled the: 

• Conversion of important historic government and other materials to digital format, 

• Creation of new government information in digital format, 

• Organization of government information in more useful ways for citizen access, 

• Change in internal institutional practice necessary to better deliver government 
information to citizens via electronic networks, 

• Increased capacity to deliver information via world wide web pages on electronic networks, 

• Ability to receive electronic government information extended to rural Oregon libraries. 

This is due to the significant progress made to date by Oregon State Library, Multnomah County 
Library, and the Oregon Historical Society toward meeting their grant objectives. PORTALS itself 
will need to make additional strides during the remaining grant period in order to achieve its own 
programmatic objectives under the grant. 

Recommendations 

PORTALS Should Re-Assess its Interest in and Responsibility for Grants of this Type 

PORTALS has a new interim Executive Director and there is a new President at Portland State 
University, a key PORTALS member. This is an opportune moment for reassessing appropriate goals 
and directions for PORTALS. Indeed, this activity is already underway. The evaluators note the 
activities of the interim Executive Director in this area as represented in the PORTALS draft strategic 
plan (Available: WWW:http://www. portals.org/plantoplan.html). 

Although overall management and organization of PORTALS is beyond the scope of this study, 
it is important to recognize that some PORTALS management issues did affect project activities. The 
components of the project for which PORTALS was responsible suffered from: 

• Inadequate PORTALS and PORTALS membership staff to complete project activities: At 
PORTALS, staff assigned to the project had other, more compelling duties; staff were not 
hired (i.e., the local evaluation assistant); plus there were staff turnovers and position 
vacancies. Staff from the PORTALS membership designated in the proposal to complete 
tasks did not, voluntary participants from member institutions were not solicited or used. 

• Lack of shared project information and status of project activities among PORTALS board, 
Council of Librarians, and PORTALS membership. At best, this encouraged lack of interest, 
at worst, lack of project information created fertile ground for rumor. 
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• Competing views as to the appropriateness and importance of the project, and specific 
project component parts, from PORTALS Board, Council of Librarians, and PORTALS 
member organizations. 

• Failure to share information among grant participants about the grant and other related 
activities of grant participants. 

• Failure to utilize expertise developed by grant participants as a result of the grant for the 
benefit of other PORTALS members or even to ask how such an asset could be utilized. 

In hindsight, it is clear that there was inadequate project staffing at PORTALS for it to conduct regular 
PORTALS business, manage the grant, and to implement the project component parts for which it was 
responsible. The degree to which PORTALS wishes to continue in the business of obtaining, managing, 
and implementing grants such as this one is a key issue that should be considered during the current 
strategic planning process recently initiated. 

PORTALS decisions about goals and direction directly affect the outcomes of the present and 
future HEA II-B grants. As part of this assessment process, the evaluators suggest the following actions 
related to the HEA II-B grants: 

• Ask current grant participants to prepare a briefing for a joint session of the PORTALS 
Board and Library Council as well as other interested members. Specifically ask the grant 
participants to discuss ways the grant benefitted their institution and to identify potential 
ways the grant recipients might further benefit PORTALS as a result of the grant. 
Disseminate written briefing materials via the PORTALS website as well as other means. 

• Ask future HEA II-B grant participants to brief the PORTALS Board and Library Council as 
well as other interested members at the same meeting suggested above on their planned use 
of their HEA-II-B grants. Specifically ask the grant participants to discuss ways the grant 
is expected to benefit their institution and to identify potential ways the grant recipients 
could contribute to PORTALS as a result of the grant. 

• Create a committee to report to the interim Executive Director, composed of PORTALS 
members charged to (1) coordinate preparations for the joint Board- Library Council 
meeting discussed above (2) capture from that meeting and discussions with grant 
participants the ways PORTALS might benefit from HEA II-B grant projects, (3) post this 
report to the PORTALS website. (4) develop plans within the PORTALS structure for 
achieving those benefits for presentation to the interim Executive Director, Executive 
Committee and/or the Library Council. Post this report to the PORTALS web site. (5) 
recommend to the interim Executive Director policies and procedures to be followed by 
PORTALS and grant recipients upon award of a PORTALS sponsored grant. 

• Present to the Board or Executive Committee a report prepared by the interim Executive 
Director containing recommendations for making use of the institutional assets created by 
the HEA II-B grants. 

• Post this report to the PORTALS web site. 
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The evaluators believe this approach will contribute to enabling PORTALS members to work together 
more effectively. Indeed, the evaluators believe the PORTALS membership will not be able to 
effectively address PORTALS present and future involvement with HEA ILB grants without these or 
similar steps. 

Ask Current Grant Participants for Ways PORTALS Might Support Grantees Better 

The new Principal Investigator for the HEA ILB grants should contact the principals in the 
current and future HEA ILB grant cycle to ask for advice on ways that PORTALS might better support 
grant participants. This will establish the new relationship in a positive light and perhaps simple 
changes may make a big difference. The outcome of this discussion should be a written report to the 
interim Executive Director. 

Reconsider Technology Choices When Planning the Next Grant or PORTALS Member Activities 

The former Project System Programmer may be the only staff member of all of the grant 
participants who is comfortable and fully conversant with UNIX software at the level necessary to 
accomplish grant objectives. Yet UNIX software and supporting hardware was at the technological 
heart of each of the grant participants proposed projects. All grant participants struggled with the 
UNIX software and its associated technology purchased for the grant, performed grant activities on 
alternative technology and software, or preferred different technology and software. Uncomfortable 
technology and software delays progress. The lack of familiarity with technology and software to be 
deployed to PORTALS members constrained programmatic choices and delayed implementation. The 
evaluators recommend that if possible the process whereby the grant technology was chosen be 
examined, not to place blame, but to learn how not to make similar choices in the future. As an 
alternative, build in training for system administrators at grant sites so that the comfort level increases 
— the technology chosen is in all other respects highly regarded. 

As a separate matter consider budgeting for an uninterruptable power supply and firewall 
software for each Sun Sparc class computer purchased. These were unanticipated expenses for OHS. 
Finally, consider advising (but not requiring) grantees involved in experimental efforts (e.g., OHS 
during this grant period) to reserve up to 25% of their technology budget until well into the grant 
period. Technologies change rapidly, ideas about how to use technologies change almost as fast. It 
would be a shame to discover what technology is really needed after budgeted monies are expended. 

Require Each Grant Participant, Including PORTALS, to Produce a Final Grant Report 

Grant participants, including PORTALS, should be required (by PORTALS) to file an end of 
funding cycle report (after 9/30/97 cycle ends). The final report of each grant participant should follow 
a format similar to the present onsite evaluation. What were the grant objectives as proposed and a 
summary of key accomplishments? What were the key events, agreements, activities, etc. in 
chronological order? Identify ways found to aid in staff development? Describe principal work 
products as a result of this grant? Identify grant dissemination efforts by each institution? What 
lessons learned? This section, if done right, could be the most helpful for the institutions involved. The 
grant prompted major if not profound changes in how grant institutions deliver information internally 
and to citizens. Capturing and beginning to understand these changes can lead to significant 
organizational improvements at each of the participating institutions. Finally, the report should give 
some attention to next steps and recommended actions each institution plans to take. 
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Seek Ways to Promote Exchange among the Grant Participants 

The onsite consultant was repeatedly struck by the similarity in some experiences among the 
participants. Donna Reed's experiences developing county and regional government web sites has much 
in common with the work of Ernest Perez (and Scott Smith and Mike Freese) in State government. Donna 
Reed and Rushton Brandis and the JumpStart libraries have much in common as they transform 
libraries into digital community information centers. All of the participants might benefit from OHS's 
experience with converting information in a range of formats into digital material accessible on the 
web. PORTALS should establish mechanisms to combine the experiences of the various grant 
participants for the good of the Portland area community. PORTALS efforts might extend to offering a 
one day conference with published proceedings showcasing the work of the grant participants. 

Seek Ways to Make PORTALS and its Grant Projects More Visible 

PORTALS was a remote entity poorly understood, to many of the grant participants visited by 
the Onsite Evaluator. PORTALS was not known to the users of products and services produced by the 
grant with whom the Onsite Evaluator came in contact. PORTALS needs to get its story out better. A 
start would be a tour of the grant participants projects by the Principal Investigator (and interim 
Executive Director perhaps). 

The work of Multnomah County Library, the Oregon Historical Society, and the Oregon State 
Library would have national interest and impact if disseminated. The evaluators suggest PORTALS 
vigorously pursue its role as grants administrator and find ways to encourage and enable the grant 
participants to disseminate their findings. 

PORTALS Should Hire the Staff to Administer Large Grants 

For better or worse, PORTALS is in the business of administering large grants. This is a 
different business than running a network or sharing databases, or coordinating collection development 
or running a consortia. PORTALS should recognize that it is in the grants administration business and 
hire the staff to manage the grants, evaluate grant participants work, publicize grant activities, scan 
for new grants for the PORTALS membership, assist with grant proposal development, disseminate 
project findings, and the other tasks required in the grants administration business. 

Consider Now How Subsequent HEA II-B Grant Segments Should Be Evaluated 

PORTALS attention to grant evaluation from the proposal stage to date , when it thought about 
evaluation at all, was minimal. Notable were PORTALS failure to hire the local onsite evaluator 
funded by the grant and failure to file the 2/97 evaluation report requested by the Evaluation 
Coordinator. Without regular evaluation, compounded by staff turnover, multiple grant partners, and 
the award's size, problems were likely to occur. The evaluators strongly suggest immediate attention be 
given to developing a mechanism for continuous evaluation of the HEA II-B grant. 

The portion of the HEA II-B grant evaluated here is the first phase of several future, multi- 
year grants awarded within one contract to PORTALS by the Department of Education. There are no 
plans to evaluate subsequent phases of this grant contract. Advanced planning for evaluation and close 
monitoring of the implementation of the resulting plan will ensure higher quality outcomes than 
otherwise possible. The evaluators recommend that PORTALS develop a plan for evaluating 
subsequent portions of the HEA II-B grant and act on it. 
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Extending Partnerships 

Participants in the grant can take considerable pride in the accomplishments produced by this 
project thus far. The work done by the Oregon State Library, Multnomah Public Library, and the 
Oregon Historical Society clearly makes a significant improvement in citizen's access to government 
(and other) information electronically. The recommendations offered in this report suggest strategies 
that can build on the work done to date. The overall goals of the project either are or are soon to be 
accomplished. 

Perhaps more important than the accomplishment of project goals is how the project provided a 
focal point for different organizations to work together successfully. To some extent, the project served 
as a catalyst that brought together a variety of organizations and agencies to provide improved public 
access to a range of electronic information resources. Moreover, the knowledge gained by project 
participants in implementing the grant is another legacy that can be used in the future. This knowledge 
and sense of partnership will provide an important base for future joint projects, coordinated resources 
sharing, and leveraging of resources for the benefit of everyone. 
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Appendix P-1 

Evaluation of Principal Sites Funded by HEA II-B Grant to PORTALS 

As part of the site evaluation the Onsite Evaluator examined existing web sites at the 
participating institutions as of May 1, 1997. These sites include: 

PORTALS WWW page. Available: WWW: http://www.portals.org/ 

Multnomah County Library. RITNet. Available: WWW: http://www.region.portland.or.us/ 

Multnomah County Library Web site. Available: WWW: http://www.multnomah.lib.or.us/ 

Oregon Historical Society WWW page. Available: WWW: http://www.ohs.org/ 

Oregon State Library. Oregon Public Library: JumpStart page. Available: WWW: 
http: / / govinfo.kerr.orst.edu /jumpstart/jump.html 

I did not cover the state library sites because I could not figure out which ones the HEA II-B grant 
funded prior to the onsite visit. 

The brief reviews and recommendations follow. In addition, there is a list of sources for 
evaluating web pages section included. Be warned this is "quick and dirty" use at your own risk. As the 
Onsite Evaluator my principal interest was in familiarizing myself with the web sites and 
organizations and not in conducting a full-blown evaluation. 
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Recommendations for the PORTALS WWW Page ( http://www.portals.org/) 

Reviewed on May 1 1997 



Recommendations: 

1) Reduce clutter, change the organization, and sharpen focus of the page. It was on clear to the 
evaluator who your principal audience(s) were. It was also not clear what PORTALS priorities are 
from the page. Some suggestions to start the process.... Reorganize opening screen around the following 
items: ( underlined words indicate links to subsequent WWW pages.} 



PORTALS is an organization of public and private institutions committed to working cooperatively 
in order to expand and enrich the information resources and services necessary for the scholarly 
research activities of people in the greater Portland metro area. 

• Welcome to PORTALS 

Includes PORTALS customer service FAQ, and reciprocal borrowing agreement . 



• PORTALS Members and Contacts 



Includes current PORTALS member list, institutional home pages, library home pages, and library 
catalog s 

• PORTALS Information Services 

Includes links to PORTALS licensed databases, Internet reference sources and pathfinders, and 
government information available via the Internet. 

• Feedback 

PORTALS looks forward to your corrections, new ideas, and comments. 

Maintained by: XXXXX XXXXXX 
Last Revised: XXX XX, XXXX 

Notes: 

a) Welcome to PORTALS subsumes the following present sections of the PORTALS opening 
www page (see above): About PORTALS, Customer Service FAQ, and Reciprocal Borrowing. I 
would add information about PORTALS staff, other public information about PORTALS (I.e., 
budget, board members and how governed, board minutes, other administrativa, reports, plans, 
project status ~ decent model for starters is state library). 

b) PORTALS Members and Contacts : subsumes PORTALS Institutions, Institutional Home Pages, 
Library Home Pages, Library Catalogs, and Contacts. 



c) PORTALS Information Services : subsumes the following present sections of the PORTALS 
opening www page (see above): PORTALS Licensed Databases, Reference Shelf, Internet 
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Information Services, Searching the World Wide Web, Government Information, 
d) Feedback : is a new section. 



2) Thoroughly proof existing WWW pages to correct typos and other unintended errors. 

E.g. Opening screen: 

"Reference Shelf 

This sections contains links to some of the most heavily used resources available via the Internet. " 
The word "sections" should be "section." 

E.g., Contacts link (listing PORTALS members) fails to list Portland State University 

3) Change to the Welcome to PORTALS screens 

a) Combine the PORTALS mission statement and customer service FAQ with the mission statement the 
first question asked and answered. Leave as is the reciprocal borrowing agreement. 

b) Eventually you could include such items as web usage statistics, what new, etc. 

4) Change to the PORTALS Members and Contacts screens 

a) Merge the information presently contained in the contacts screen with the member list 

5) Change to the PORTALS Information Services screens 

5A) New PORTALS Information Screen appears as follows: 

Reference sources and pathfinders 
PORTALS licensed databases 
Government information 

5B) Merge prior Reference Shelf, Internet Information Services, Searching the World Wide 
Web pages into a new Reference Sources and Pathfinders Page 

Scope: {What exactly is your criteria for including sources here, who is your audience. 

Pathfinders and Guides (present Subject Directories): Agriculture I Anthropology I Biosciences I 
Business (General) I Business (International) I Children I Classical Literature I Diversions I Education 
I Introduction to the Internet (present Information About the Internet) I Job/Career Information I 
Literature I Plays I Sociology I Weather I 

(Merge Business and Financial Information in with Business (General and International)) 

Quick Reference 



Search Tools 
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Directories: Libraries, Museums, Research Centers, Societies 
Library Catalogs 
Electronic publications 
Maps 

{The above reflects what you have on hand, note I recommend dropping from this section sources cited 
elsewhere (licensed databases, government sources, community networks, etc. Potential new categories 
include: Librarian Professional Development, News, Local Companies, Build Your Own Web Page, etc. 
Don't include if categories can not be maintained or are out of scope. Merge Searching the World Wide 
Web, at present it is duplicative (of, for example, search tools proposed above)} 

5C) Change the PORTALS Licensed Databases page as follows: 

Screen 1: 

PORTALS Licensed Databases 

License Restrictions: PORTALS databases are licensed from copyright holding vendors and 
publishers for use exclusively by faculty, staff, students, and users of consortium members. All 
others can not access these databases via PORTALS. Please understand that the restriction is a 
result of the limitations of our contracts with the database vendor. 

Problems: Student, faculty, or staff of academic members of PORTALS should contact their 
institution for assistance. For contact information click here . Multnomah County Library card 
holders call the Library at 248-5223 for the number of their dial access port. 

Index: {As is} 

I AIDSLINE (AIDS Information Online) I BIOSIS Previews I Business Abstracts I CINAHL 
(Nursing & Allied Health) I Dissertation Abstracts I El CompendexWeb I ERIC I 
HealthSTAR I MEDLINE I MLA Bibliography I Newspaper Abstracts I Periodical Abstracts 
Research II I PsychINFO I UnCover I WORLDCAT I 

New Screen 2: 

List of Databases [ok as is] 

{Note, while I have altered the wording and would like to shorten the text overall, my principal 
suggestions are two: 

a) Will users understand what "port of access is?" 

b) Move the license restriction to the same screen as the index to the databases.} 



5D) New PORTALS Government Information Screen is present Government Information screen. 
To be consistent why not include Portland under Local Government as follows: 

Local Government Information 

Links to local government and community networks in Oregon including the Portland Metro area . Will 
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eventually include similar pages with links for Alaska, Idaho, Montana, and Washington. 

In a similar fashion: 

County Government Information 

Links to county government web pages in Oregon including Multnomah County . Will eventually include 
additional pages with similar links for Alaska, Idaho, Montana, and Washington. 

6) Introduce Feedback section. 

a) A simple place to start is to include a "mailto" form in addition to the existing opportunity to get in 
touch with the webmaster. Indicate that comments are welcome! Prompt the users as to why they 
might want to offer feedback (what types of topics) and clearly indicate that if you click here your 
comments will receive attention say within 24 hours. Feedback prompts might be useful at more than 
one location at the website. Later might include mailtos to individual staff if that is deemed 
desirable. You could also solicit feedback on specific topics, etc. using a cgi form. The maito only works 
if you have an email address and your browser is properly configured. Thus most users sitting at public 
terminals cannot use this option. Cgi forms, while a bit harder to program will work if the user does not 
have an e-mail account and also if the users is using a public terminal. 
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Recommendations for the Multnomah County Library's RITNet site 
(http://www.region.portland.or.us/) 

Reviewed May 1st 1997 



General Issues 

Is there some logic to the listing of the various menu items, would an alphabetical listing be better? 
Specific Items 

The following recommendations are keyed to the RITNet menu items. 

1) Search RITNet 

It might be better to separate searching the RITNet site from searching the rest of the world. 

2) Elections 

I know it was a big hit but why keep now? 

3) Citizens Involvement I did not get the point of this section. Many of the other menu items you list are 
also in the business of promoting citizen involvement (e.g., Neighborhood Associations, non profits) 
why isolate what appear to be libraries in this menu item. 

4) Maps and We Are Here 

a) Why not use instead: 

MapBlaster. Available: WWW: http://www.mapblast.com/ 

For starters try specifically: 

http:/ /www. vicinity. com/ yt.hm?FAM=mapblast&CMD=GEO&SEC=find&IC=0%3A0%3A 
5&IC%3A=Portland&AD2=&AD3=Portland%2C+OR 

Look at my html code at: http://web.syr.edU/-jryan/infopro/gov.html#Maps 

b) Then consider adding: 

Kindred, Darrell. How far is it? Available: WWW: 

c) I was not sure what We are here was intended to mean. You might also want to include your street 
address, phone, fax, and solicit e-mail feedback. 

5) Parks 

Why list the url for: Crown Point State Park? Vista House? http://www.vistahouse.com/ 



6) Arts and Entertainment 
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Check html coding here, when I clicked on museums or university events I was sent to the main menu. 

7) Creation date is of interest but so is most recent page revision. 

8) See comments regarding County Government site next 



Ryan/ McClure Onsite Evaluation Report: Citizen Access to Government and Other Information August 12, 1997 



Recommendations for the Multnomah County Library's Multnomah County's Web site 

(http://www.multnomah.lib.or.us/) 

Reviewed May 1st 1997 



General Issues 

1) Is there some logic to the listing of the various menu items, would an alphabetical listing be better? 

2) Why not a menu item for libraries or one under non-profits for the libraries in the county 

3) Next logical step is for a forms based application procedure where citizens can file for jobs, benefits, 
make complaints, etc. Directly on the Internet. For one example, see Maricopa County, AZ: 
http://ww2.maricopa.gov/feedback/services.html-ssi and 
http://ww2.maricopa.gov/feedback/feedback.html-ssi 

4) Other additions include subject based access based on frequently requested citizen information needs. 
Even a simple subject index might help (Montgomery County: MD: 
http://www.co.mo.md.us/services/services.htm) 

5) A broad next step is to re-think the page in terms of major audiences needs. Who are the audiences: 
visitors, the press, planners, increasing citizen involvement, relation to politics and electioneering, 
benefits? What are their needs? One can start by comparing the present site to other county sites across 
the nation. However an in depth study of the local county needs is necessary. 

6) Consider the need for a legal disclaimer (e.g., Dade County, FL: http://lola.co.lee.fl.us/legal.htm or 
more elaborate: Hennepin County, MN: http://www.co.hennepin.mn.us/wservice.html) 

7) Other sites you might want to look at : 

frequently asked questions (Hennepin County, MN: http://www.co.hennepin.mn.us/wfaqs.html), 
phone directory [frequently called numbers] (Hennepin County, MN: 

http://www.co.hennepin.mn.us/wfreqnos.html), contracts and purchasing (Dade County, FL: 
http://lola.co.lee.fl.us/purcmenu.htm Orange County: http://www.citizens- 
first.co.orange.fl.us/divisions/f&hrd/purchase/hotline.htm), county documents index (Hennepin 
County, MN: http://www.co.hennepin.mn.us/wdocs.html), speakers bureau (Dade County, FL: 
http://lola.co.lee.fl.us/speakers.htm), guest book (Montgomery County, MD: 
http://www.co.mo.md.us/cgi/comogest.cgi), surveys (Wayne County, MI: 

http://www.waynecounty.com/survey.html and http://www.waynecounty.com/response.html). 

Simple but effective opening page design: Johnson County, KS: http://www.jocoks.com/ 

Notion of economic development, tourism, visitor services 
Specific Items 

The following recommendations are keyed to the County Web site menu items. 

1) Agencies and in general, check for typos like this: ’’Regional Drug Initiative ” 

2) Demographics and weather: Perhaps trying to do too much here?! I would separate out into four 
separate menu items: basic facts, history (County Inventory and other sources), weather, demographics 
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(might want to include census information), etc. For an example of possible demographics information to 
include see Alameda, CA Economic and Demographic Information: http://www.alameda.org/ 

3) Maps Geographic Information Systems are a area of rapid growth and intense interest among local 
government officials. Watch for local initiatives /interest in getting GIS information on the Internet. 

5) What exactly does the county cover (as compared to cities and state)? Does the county deal with 
Courts, Education, have a budget, issue, Fines, Taxes, Permits, Licenses, Payments, Health and Welfare 
Information, Law Enforcement, Public Assistance, Public Works, Zoning, Planning. {I know now you are 
working to resolve this with Celia Heron and others....) 
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Recommendations for the Oregon Historical Society WWW page (http://www.ohs.org/) 

Reviewed May 1st 1997 

1) Add a text only option for presentation of the page. While I am not fully conversant with the Oregon 
situation, my sense is that a significant portion of the likely Oregon institutional and citizen users will 
access the page via a dialup connection. Graphics, frames, Java, all contribute to reducing accessibility, 
or usability for the dialup user. I am not suggesting you drop graphics, frames, Java scripts, and other 
state-of-the-art methods of presenting the site. I am suggesting providing multiple ways of accessing 
the sites content where possible and appropriate. 

2) Consider adding links to other state historical societies. I have compiled a list of them which you 
are free to use see: http:/ /web. syr.edu/~jryan/portals/histsoc.html 

3) At present not much access to the full range of digitized images processed for the grant. No doubt this 
is because processing of these images is not complete. It is not clear how OHS will make available all 
of these images to the WWW public. 

4) Focus on adding content and not (only) images: Perhaps in OHS’s haste to show of its work related to 
the grant it has over-emphasized graphic images. It might be useful to go back to Todd's original home 
page (and the one on PORTALS) and get some of these content areas in. Also taking a look at related 
historical sites (see 2 above) might offer some insight. I remain in favor of an OHS staff directory with 
address, phone, fax, and e-mail links. 
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Recommendations for Oregon State Library 
Oregon Public Library Home Page (JumpStart Project) 
(http://govinfo.ken\orstedu/jumpstart/jump.html) 

Reviewed May 1st 1997 

Menu items include: art, books, business and employment, grants and funding, health, house and 
home, internet and computers, kids' Sites, movies, radio and television, music, native american 
resources, news, phone directories, maps, zip codes, reference, libraries on the web, science, sports, tax 
forms and publications, time, travel, and weather. In addition, there is information on educational 
sites, government & community information, library links, internet searching (using subject directories, 
keywords and search for people and e-mail addresses) and further information on the JumpStart library 
project. In May 1997, the maintenance of this page was taken over by M.L.S. students at Emporia State 
University (Kansas). 

The Onsite Evaluator would have liked to examine the Jumpstart Participating Librarians 
Support Page section but was not able to obtain permission. My initial reaction to this page was that it 
was very well done. Subsequent onsite discussions with JumpStart librarians and users confirm my 
initial impression. My concerns are really longer term, for example, 

• How will you obtain feedback to keep in touch with changing interests. You do not ask for 
feedback on the web page. I suspect you do not track the types of questions people are asking 
of your most popular menu item: search engines. My sense of the audiences at the JumpStart 
libraries is that you would want to keep a close eye on current interests. 

• While I could not look at the JumpStart participant page, I wonder about opportunities do 
deliver courseware and continuing education to the various users of the web page? 

• One of the librarians surveyed had not heard about the JumpStart listserv, another did not 
use it. I am wondering (1) should the list be re-advertised (2) could its content be beefed up 
to attract users, or (3) should it be dropped because rural librarians in Oregon use other 
listservs. 

• One of the most heavily used or requested (where they do not allow use) services by 
teenagers is chat areas, MOOS/MUDS etc. A chief concern among local Library Boards is 
what might go on in these "dens of iniquity." Is there a possibility of providing moderated 
chat areas satisfying the concerns of adults while not losing what makes this form of 
communication so appealing to teenagers? 

• Is it possible to increase the interactivity of the site by using simply CGI forms to (1) allow 
for reference requests (to be handled by the State Library Reference Link program or 
JumpStart librarians in a round-robin fashion) (2) permit conversations on hot topics, a CGI 
form takes the comment and dumps it to some type of threaded-mail program so that 
participants could track the "conversation." (the CGI approach is suggested because most 
library users cannot make use of the e-mail feature, either because the do not have an e- 
mail account or the library does not permit sending of e-mail using the public machine. 

In any case the present service his highly valued by the public libraries I visited, these are some quick 
additional suggestions. 
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Internet-based Sources for Evaluating WWW Pages 



Compiled by 
Joe Ryan 

<jryan@mailbox.syr.edu> 



General Style Guides 

Advice to Christine. Available: WWW: http:// www.el-dorado.ca.us/ -advice 

Advice on home page and net graphic design in the form of memos. 

Australian National University. Internet quality guidelines. Available: WWW: 
http: / / coombs.anu.edu.au/SpecialProj / QLTY/ QltyHome.html 

Bertot, John Carlo, McClure, Charles R., Moen, William E. and Rubin, Jeffrey. (1997). Web usage 
statistics: Measurement issues and analytical techniques. Available: WWW: 
http://research.umbc.edu/~bertot/epa.stats.html 

Ciolek, T. Matthew. (1994-1995). Quality, guidelines & standards for internet information resources. 
Australian National University. Available: WWW: 
http://coombs.anu.edu.au/SpecialProj/QLTY/QltyHome.html 

Cohen, J. K. (1995). Elements of HTML style. Available: WWW: 
http:/ /bookweb.cwis.uci.edu:8042/Staff/StyleGuide.html 

Design of HTML for those with Disabilities. Available: WWW: 
http://www.trace.wisc.edu/HTMLgide/ 

Herrick, Michael. Design considerations. Available: WWW: 
http://www.matterform.com/mf/hypermedia/hypermediahome2.html 

Horton, William, Taylor, Lee, Ignacio, Arthur, & Hoft, Nancy L. (1996). The web page design cookbook. 
NY: John Wiley. 

HTML writer's Guild. Available: WWW: http://www.mindspring.com/guild 

Hurley, Jim. Articles on webspace design. Available: WWW: 
http://www.webcom.com/~hurleyj/article/index.html 

NCSA. HTML Style sheet. Available: WWW: 
http://www.ncsa.uiuc.edu/Pubs/StyleSheet/NCSAStyleSheet.html 

Netscape, Inc. Creating high impact documents. Available: WWW: 
http:/ / www.netscape.com/home/services_docs/impact_docs/ 



Quinn, Christine. From grass roots to corporate image: The maturation of the web. Available: WWW: 
http://www.ncsa.uiuc.edu/SDG/IT94/Proceedings/Campus.Infosys/quinn/quinn.html 
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Discusses Stanford University's evolving efforts to develop a "corporate" image for its web pages. 

Richmond, Alan. (1994). A basic HTML style guide. NASA, Goddard Space Flight Center. Available: 
WWW: http://guinan.gsfc.nasa.gov/Style.html 

Richmond, Alan. CyberWebs Top Tips for Web Authoring. Available: WWW: 
http://WWW.Charm.Net/~web/Style/ WWW: 
http://WWW.Stars.com/Vlib/Providers/Style.html 

Sanders, Tony. Bad style page. Available: WWW: http://www.earth.com/bad-style 

Set it up ergonomically! Available: WWW: 
http://www.best.com/~dsiegel/vestibule/set_netscape.html 

Signall, Karl. Do’s and Don’ts. Available: WWW: http://millkem.com/do-dont.html 

Tillman, Hope N. (1997, April 17). Evaluating quality on the net. Available: WWW: 
http://www.tiac.net/users/hope/findqual.html 

Tilton, James (Eric). Style guide by counter example for HTML. Available: WWW: 
http://www.willamette.edu/html-composition/strict-html.html WWW: 
http://www.cs.cmu.edu/~tilt/cgh/ 

Top ten things not to do on a web page. Available: WWW: 
http: / /ee.stanf ord.edu / eecns / w w w / donts .html 

Top ten ways to tell if you have a sucky home page. Available: WWW: 
http://www.wintemet.com/~jmg/topten.html 

Web wonks tips for writers and designers. Available: WWW: 
http://www.best.com/~dsiegel/tips/tips_home.html 

Werbach, Kevin. What makes a good home page. Available: WWW: 
http://www.access.digex.net/~werbach/page_design.html 

W30. Design Issues. Available: WWW: 

http://www.w3.org/hypertext/WWW/DesignIssues/Overview.html 

Yale University. Center for Advanced Instructional Media. WWW style manual. Available: WWW: 
http://info.med.yale.edu/caim/StyleManual_Top.html 

Best Sites 

There are a range of different types of sites evaluated on a range of criteria (none considering the local 
context, stakeholders, etc.). 

Exemplary WWW Sites. Available: WWW: http://www.sei.cmu.edu/tech/demo/good_sites.html 

Global Network Navigator. Best of the web. Available: WWW: 
http://gnn.com/gnn/wic/best.toc.html 
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Point Communications. Best of the Web. Available: WWW: http://pointcom.com/jpegs/topsites/ 

Yahoo: Computers and Intemet:Intemet:World Wide Web:Best of the Web. Available: WWW: 
http://www.yahoo.com/Computers_and_Intemet/Intemet/World_Wide_Web/Best_of_the_Web/ 

Yahoo: Computers and Intemet:Intemet:World Wide Web:Best of the Web:Not Really the Best. 
Available: WWW: 

http://www.yahoo.com/Computers_and_Intemet/Intemet/World_Wide_Web/Best_of_the_Web 
/ Sites_of_the_Day / 

Yahoo: Computers and Intemet:Intemet:World Wide Web:Best of the Web: Sites of the Day. 
Available: WWW: 

http://www.yahoo.com/Computers_and_Intemet/Internet/World_Wide_Web/Best_of_the_Web 
/ Sites_of_the_Day / 
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Appendix P-2 

Schedule of Onsite Evaluation Interviews 

Tuesday June 3 1997 
Arrive in Portland 

4PM Jim Kopp, PORTALS interim Executive Director, PORTALS office — review evaluation plan. 
Wednesday June 4 1997 

10AM Introductory meeting with Donna Reed, MCL 

1 PM Meet with PORTALS Board, Pacific Grove University Forest Grove, OR 

Thursday June 5 1997 

10AM - 5PM Met with OHS 

First met with the majority of OHS involved with the project in a group introduction and 
discussion. Then spent the rest of the day meeting with Elizabeth Winroth, Maps Librarian, Laura 
Ayling, Map cataloger, Jim Labosier, Maps Assistant, and Mandy York, Maps and Photographs 
Cataloger; Barbara Abrams, Project Navigator and Deputy Director for Operations and Programs; 

Steve Hallberg, Chief Cataloger. 

Friday June 6 1997 

Meet with State Library participants 

8:30 Ernest Perez, State Library, Room 101 General introduction, agenda review 
9:00 Jim Scheppke, General meeting & discussion 

10:00 Ernest Perez OSL activities, user experience, cooperative agency efforts 

11:30 Scott Smith, Information Systems Coordinator, Strategic Planning & Review, Department of 
Administrative Services and Mike Freese, State Printer 

2:00 Roy Tumbugh, State Archivist, State Archives regarding Oregon Administrative Rules, Echoes 
of Oregon History 

3:00 Rushton Brandis Review of Jumpstart project activities 
Monday June 9 1997 

10:30-1:30 Donna Reed, RITNet, Multnomah County Library 

5 PM Telephone interview with Dennis Gilbert, who recently resigned as Project System 

Programmer 
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Tuesday June 10 1997 
Meet with OHS 

9AM Kris White, Director of Archival Collections 

10 Marsha Williams, Director of Museum Collections in charge or artifact digitalization 

1PM Chris Bostic, MIS Director and Communications Manager and Dwight Patterson, MIS Senior 
Engineer 

2PM James Strassmaier, Oral Historian 

2:30PM Sue Seyl, Director of Image Collections, Richard Yost, Imaging Technician, Mandy York, maps 
and photographs cataloger, Elizabeth Winroth, Maps Librarian, and Mikki Tint, Assistant 
Photographs Librarian, Evan Schneider, Photographer all from the Photography Department. 

Wednesday June 11 1997 

Meet with OHS 

9AM Todd Welch, Project Archivist 

1PM Chet Orliff, Oregon Historical Society Executive Director 
Thursday June 12 1997 

3PM Examine OHS Web Task Force Minutes and other pertinent documentation 
5PM Karen Starr, PORTALS, Director of Network Information 
Friday June 13 1997 

10AM Seaside Public Library, Reita Fackerell, Director and Paula Clark Project Manager 
Monday June 16 1997 

9AM Driftwood Public Library, Lincoln City, 801 SW Highway 101, Susan J. Jenkins, Assistant Library 
Director and Project Manager, and Cataloger and Systems Administrator 

Tuesday June 17 1997 

Went from Newport to Corvallis in hopes of meeting Charlene Grass but could not connect 
Thursday June 19 1997 

10AM Bill McCabe, Washington County Government webmaster 
1-3PM RITNet Advisory Council at METRO, MLK Bvld. 

3-5 Celia Heron, Director of Information and Referral and Webmaster City of Portland Web page 
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Friday, June 20 1997 

1PM Oregon State University, Valley Library, JumpStart Training and Technical Support staff. 
Government Information Sharing Project, Christine Grasse, Project Manager; Stephen Mosley 
Research Assistant, Information Services (the technical support coordinator); Judy Cross, 
Government Documents Librarian; Cheryl Middleton, Life Sciences Librarian and Kerry Otto. 

Wednesday, June 25 1997 

1PM Exit interview with Jim Kopp, PORTALS interim Executive Director 




ERIC 
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Appendix P-3 



Contacts for the Onsite Evaluation Visit 



PORTALS 

P.O. BOX 1151, Portland, OR 97207-1151 

Phone: (503) 725-3361, 725-5794 

Fax: (503) 725-4625 

WWW: http://www.portals.org 

Contacts: 

Jim Kopp, Ph.D. <kopp@portals.org> interim Executive Director, Phone: (503) 725-5992 Fax: (503) 725- 
4625 



Karen Starr <kstarr@portals.org> <kstarr@wln.com>. Director of Network Information (503) 725- 
5049; FAX (503) 725-4625 

Resigned 

Howard McGinn, Executive Director 

Dennis Gilbert <dennis@gst.world.net> Project Systems Programmer 
Portland State University (PSU) 

Marjorie Enneking, Associate Vice Provost (503) 725-3416 
Millard Johnson, Director Network Development (503) 725-5049 
Gwenn Newborg, PSU Business /documents Librarian 
Judy Ngai, Fiscal on grant (503) 725-5084 

Sue Reggiani <reggiani@lib.pdx.edu> (503) 725-4576; (503) 725-4524 Branford Price Millar Library, 
Portland State University, PO Box 1151, Portland, OR 97207-1151 

Muntnomah County Library 
205 NE Russell Street 
Portland, OR 97212-3708 
Phone: (503) 248-5123 

WWW: http://www.multnomah.lib.or.us/ 

Jeanne Goodrich <jeanneg@nethost.multnomah.lib.or.us> Phone: (503) 248-5492; Fax: (503) 248-5441 
Deputy Director/ Program Administrator, Multnomah County Library 

Donna Reed <donnare@nethost.multnomah.lib.or.us> <donnare@pub.multnomah.lib.or.us> (503) 248- 
5238, Fax: (503) 248-5226 Community Information System Program Specialist 

Brian Williams <brianw@nethost.multnomah.lib.or.us>; Fax: (503) 248-5226 Manager, Automated 
Systems, Multnomah County Library, 801 SW 10th, Portland, OR 97205-2597 
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Oregon Historical Society 
1200 SW Park Avenue 
Portland, OR 97205-2483 
Phone: (503) 222-1741 
Fax: (503) 219-2040 
(503) 221-2035 

Barbara Abrams <Barbaraa@ohs.org>, Project Navigator (Deputy Director for Operations and 
Programs) 

James R. Blackaby <jblackaby@ushmm.org> Visiting faculty. University of Victoria, President of 

J.R. Blackaby 

Christine Bostick <chrisb@ohs.org> (503) 306-5217, MIS Director (computers, network) and 
Communications Manager (phone, fax, pagers) 

Steve Hallberg, Chief Cataloger 

Richard Jost <richardj@ohs.org> (503) 306-5252, Imaging Technician 
Jim Labosier, Maps Assistant 

John Mead <Johnm@ohs.org> Director of Reference and Research Collections 
Chet Orloff <cheto@ohs.org> (503) 306-5201, Executive Director 
Dwight Patterson, MIS Senior Engineer 

Beth Sandore <sandore@uiuc.edu> Coordinator for Imaging Projects, Associate Professor of Library 

Administration, University of Illinois 

Evan Schneider, Photographer 

Sue Seyl <susans@ohs.org>, (503) 306-5250; Director of Image Collections (Photograph Librarian) 

James Strassmaier Oral Historian 

Mikki Tint Assistant Photographs Librarian 

Todd Welch <toddw@ohs.org>, (503) 306-5247; Project Archivist (Manuscripts) 

Kris White <krisw@ohs.org> (503) 306-5247, Director of Archival Collections 
Marsha Williams <Marsham@ohs.org>, Director of Museum Collections 
Elizabeth Winroth Maps Librarian 
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Oregon State Library 
State Library Building 
250 Winter St NE 
Salem, OR 97319-0640 
Phone: (503) 378-4243 
Fax: (503) 588-7119 

Merrialyce Blanchard <merrialyce.k.blanchard@state.or.us> (503) 378-4198 ext. 244, Consultant, 
Oregon Index 

Rushton Brandis <rushton.g.brandis@state.or.us> (503) 378-2112 x224 Network Development 
Consultant, Library Development Services Division, Key Liason 

Mike Freese, State Printer, Printing Plant, OR Dept of Administrative Services, 550 Airport Road SE, 
Salem, OR 97310 

Arturo J. Guillen <Arturo.J.GUILLEN@state.or.us> Assistant to Automated Systems Administrator 
Vicki J. Howe <Vicki.J.HOWE@state.or.us>, Oregon Index indexer 

Ernest Perez <emest.r.perez@state.or.us> (503) 378-4243, ext. 257 Automated Systems Administrator, 
Key Liason 

Jim Scheppke <jim.b.scheppke@state.or.us>, State Librarian 

Scott Smith, Information Systems Coordinator, Strategic Planning & Review, Department of 
Administrative Services, 955 Center St. NE, Rm 470, Salem OR 97310 

Roy C. Tumbaugh, State Archivist, Oregon State Archives, 800 Summer St NE, Salem OR 97310 
JumpStart Libraries 

Driftwood Public Library, Lincoln Square, 801 SW Highway 101, Suite 201, P.O. Box 50, Lincoln City, 
OR 97367-2720 (541) 996-1253, 996-2277 Fax: (541) 996-1262 Susan J. Jenkins <sjenkins@wcn.net>, 
Assistant Library Director and Project Manager; Yueh-lin Chen, Cataloger and systems person; Patricia 
Heringer, Director 

Seaside Public Library, 60 North Roosevelt Bvld., Seaside, OR 97138-6887; Reita Fackerell, Director 
(503) 738-6742 <rfackere@ednet.osl.or.gov>, Paula Clark Project Manager Fax: (503) 738-5514 

Government Information Sharing Project 

Judy Cross <crossj@ccmail.orst.edu>, Government Documents Librarian, Oregon State University, 

Valley Library, Corvallis, OR 97331-4501 

Charlene Grass <grassc@ccmail.or.st.edu>. Associate University Librarian for Technical Services, 
JumpStart and Government Information Sharing Project Manager, Oregon State University, 421C Valley 
Library, Corvallis, OR 97331-4501 Phone: (541) 737-7302 

Cheryl Middleton <middletc@ccmail.orst.edu>. Life Sciences Librarian, Oregon State University, 
Valley Library, Corvallis, OR 97331-4501 
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Stephen Mosley <mosleys@ccmail.orst.edu>. Research Assistant, Information Services, Oregon State 
University, 121 Valley Library, Corvallis, OR 97331-4501 Phone: (541) 737-4514 Fax: (541) 737-3453 

Department of Education 

Christina Dunn <cdunn@inet.ed.gov> Director, Discretionary Programs Division, Library Programs 
Office, Department of Education, Room 300, 555 New Jersey Avenue NW, Washington, DC 20208-5571, 
Phone: (202) 219-2299 

Others Participants 

Bit by Bit Computer 9203 SWNimbus Drive Beaverton (Mike) (503) 520-0218 (800) 248-2924 
Celia Heron, City of Portland <cheron@ci.portland.or.us> (503) 823-3044 

Bill McCabe Washington County <bill_mccabe@co. Washington. or.us> (503) 681-2830, 6488721; 155 
North 1st Avenue, Hillsboro, OR 

Chuck McClure <cmcclure@mailbox.syr.edu> Evaluation Consultant (315) 443-2748 
Joe Ryan <jryan@maibox.syr.edu> Onsite Evaluator 
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PORTALS EVALUATION APPROACH 

DRAFT 

Charles R. McClure 
February 19, 1996 



Introduction 



The evaluation design for the Portals’ project has been developed in light of the 
original proposal, discussions with project participants, and changes that have 
occurred with Portals and with proposed project activities since the project was 
funded by the U.S. Department of Education. The environment in which this project 
operates is dynamic and information technology is constantly changing. Thus, some 
modifications with both the project activities and the evaluation approach have been 
done and are represented in this document. 

The original objectives of the project are: 

1. Design a strong modular architecture rooted in a client-server approach and 
distributed processing technology that accommodates remote site access and 
limitless growth in the development of new information modules and services. 

2. Develop m-otorvne information modules for delivering electronic Federal, state 
agency, local agency, and state and local archival and historical information 
and services. 

3. Test a high band- width network technology and multi-media display capability 
that is easy to use and best meets user information needs. 

4. Provide users with a range of assistance and support in using the system and 
information/services programs developed. 

5. Perform an ongoing evaluation that measures our success at achieving project 
objectives, assesses the program and services modules, provides summative 
ifrfomicition to gauge the overall success of the project, and formative 
information to improve performance during the project. 

The original proposal described aspects of the evaluation that included the following 
objectives: 

• Assess the degree to which the project programs accomplished stated 
objectives as outlined in the proposal. 

• Identify factors that contributed to the success and the limitations of the 
services and programs in particular settings. 

• Obtain assessments from the users of the programs as to: 
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- the appropriateness of the services provided 

- the ease of use of the services 

- the timeliness and usefulness of the information provided 

- the "help" and "support" provided to the users in their use 

of the programs and services 

- suggestions to improve the programs and its services. 

• Compare the relative costs of establishing and operating the programs as 
compared to the benefits resulting to the users. 

• Assess the appropriateness of the Federal information resources used in the 
project. 

• Determine the success of project collaboration efforts among the organizations 
participating in the project and in the development and operation of the 
programs. 

• Assess the reliability of the network and the technical support for the services 
and programs. 

The original project proposal noted that “the evaluation objectives may be identified 
as the project unfolds and actual methodology is developed. But overall, it is essential 
to learn which factors contribute to the success of the statewide network, what types 
of services and programs are most successful for what types of audiences, and what 
types of information services and products best meet user information needs.” 

Throughout the evaluation, six criteria will be used to shape the evaluation and 
provide a basis for addressing the evaluation objectives outlined above: 

• Extensiveness : this is a measure of the amount or extent to which the 
services are provided, e.g., the number and types of people using a particular 
service. 

• Efficiency : this is a measure of the cost or resources required to provide the 
service, e.g., cost per service transaction. 

• Effectiveness : an effectiveness measure is one that focuses attention on "how 
well" or the quality with which a service or activity is done, e.g., percentage of 
transactions that satisfactorily met user information needs. 

• Impact : an impact measure focuses attention on the benefit or result of the 
service or activity, e.g., the degree to which using a particular service 
empowered the user to resolve other problems or improved his/her quality of 
life. 
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• Usefulness : this is a measure of appropriateness, that is, the degree to which 
the services are useful or appropriate for the individual user, e.g., percentage of 
services of interest to different types of user audiences. 

• Services Quality : this is a measure of the degree to which the project’s 

services and programs are accurate, timely, and are presented to users in an 
effective m synjet*-, ergr users’- assessment of the accuracy of the information 
presented on the community net. ; 

There are six key areas for project evaluation. The first is accomplishment of overall 
project objectives and project level services/activities. The remaining five are and an 
evaluation of each of the five service modules described in the original proposal. 

It should be emphasized that the evaluation component focuses on the uses, 
impacts, strengths and weaknesses of the activities in the project modules. This 
| evaluation does not assess the appropriateness of the technology used/purchased for 
the project, the degree to which the technology itself contributed to accomplishing 
project goals and objectives, or the effectiveness and costs of the technology. 



The Overall Project 

There are a number of different evaluation areas that will be addressed in 
reviewing the project. Each of these areas are discussed briefly in this section. 

Traffic Counts 

Each of the following modules anticipates placing a number of services and 
resources in an online format that can be accessed directly via the Portals Web site. 
Each of the individual services made available will have traffic counts that provide: 

• The number of times the resources or services are “hit” by online visitors. 

• The duration of the visit to that particular resource or service. 

• An indication of the domain associated with the user’s login. 

Portals systems operations will be responsible for implementing appropriate “sniffer” 
software on the main router to produce traffic and use logs which then will be 
analyzed by the evaluation study team. Key items for analysis are the growth or 
decline in the use of specific services; which services were accessed more than other 
services; and patterns of use and access over the period of the project. 
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Project Team Time Commitment 

Beginning January, 1996, all members of the project team [include list of names of 
those individuals here] will submit a monthly report to Portals indicating the number 
of hours they committed to project activities. Portals will maintain these logs on a 
database for analysis by the study team. This aspect of the assessment should allow 
the evaluation team to assess the degree to which leveraging of resources occurs, e.g., 
the project award resulted in X hours of in-kind contribution from others to the 
project. 



Before/After Services and System Configuration. 

The evaluation should be able to describe the degree to which existing services and 
system configurations changed to that at the conclusion of the project. Coordinators 
for each of the modules (and the Portals Sysop) are responsible for describing as of 
January 1, 1996 the type, extent, and range of services currently being provided; the 
current configuration of the system; and any other descriptive or statistical detail 
describing what is in place January, 1996 as a basis for then being able to show the 
differences that have occurred by the end of the grant. Module coordinators (as well 
as the Portals Sysop?) then can describe the services and configuration at the end of 
the project, 



Target Audiences 

The proposal indicates that the target audiences for the project are virtually 
everyone. In designing the service within an individual modules it is essential that if 
the target audience is K-12 students and teachers that it be so-stated. Overall, the 
project should identify the primary and secondary audiences for the various services 
to be developed. 



Issues of Administrative Policy 

Throughout this project it is likely that a number of policy issues in terms of who 
has access, who can add information to a service, how the services will be updated 
and by whom, privacy and security concerns, etc. etc. will have to be identified and 
resolved. These policy issues should cut across the various participants in the project 
so that they are mutually supportive. A set of " Policy Guidelines" for using the 
services will need to be developed and maintained at Portals. The evaluation study 
team will review the policy statements regularly as they evolve. 
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Project Management 

The evaluation study team will regularly review the management of the project in 
terms of criteria such as: meeting the scheduled project deadlines, coordination and 
cooperation across members, managing changes in tasking, etc. Specific evaluation 
techniques include: 

• Comparing scheduled versus actual task accomplishments. 

• Review of monthly progress reports from each of the module coordinators (to 
be submitted to Portals project director), to begin with the month of January, 
1996. 

• Administering surveys to key project personnel obtaining their assessment of 
the success with which the project is being managed and implemented. 

Such evaluation information will be obtained every four months as feedback to 
Portals management staff. 



Replication/Dissemination 

A key component of the original proposal was to develop access to services and 
resources in such a way as that they could be easily replicated by other organizations 
around the country. Evaluation efforts here include: 

• Obtaining assessments from advisory committee members of the degree to 
which services and information resources could be mounted in their community 
based on the information provided by Portals. 

• Maintaining and providing appropriate documentation and information to other 
organizations so that they can determine the degree to which the various 
services might be used in their particular setting. 

• The number and types of project briefings, presentations at conferences, 
written papers and reports, and other items are developed and made available 
to the larger professional community. 

This evaluation criterion suggests the importance of the coordinators for each of the 
modules to document their project activities, develop written manuals and guides to 
assist other organizations in the implementation of similar efforts, and to make 
presentations and write papers for the larger professional community. 
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Critical Success Factors 

Throughout the various modules and services that are designed and implemented, 
the evaluation will identify “critical success factors” that contributed to the overall 
success of that particular service. A critical success factor is one that had it not 
occurred, the service would have failed or would not otherwise been completed 
successfully. Critical success factors may be related to human, technical, 
organizational, political, economic, or other aspects of the project. Surveys, focus 
groups, and/or individual/group interviews with project personnel will identify these 
critical success factors. 



Publicity and Marketing 

Portals will be responsible for developing some type of marketing plan. Records 
will need to be maintained as to the various types of advertisements, marketing 
efforts, or other types of publicity that inform the public about the project. Surveys 
can be conducted to determine the degree to which the population in Oregon is 
knowledgeable about Portals and the degree to which public knowledge increases over 
the length of the project. Also of interest is to determine the population segments 
that are not aware of the project and to determine why they are not aware of project 
services. 



Federal Information Module 



The original objectives for this module are listed below. Due to the rapid increase 
in the availability of Federal electronic information and Federal Web sites, these 
objectives may be modified. 

Objectives: 

• Develop a WWW and Lynx structure for accessing federal and other 
government information. 

• Mount online, Census data, National Trade Data Bank, National 
Environmental, Sociological and Economic Data Banks, and Regional 
Information Resources. 

• Deploy a Geographic Information System to present data in a user oriented 
graphical format. 

• Review user satisfaction and refine the system to conform to user needs. 



6 



O 

ERIC 



148 



McClure 



PORTALS Evaluation Approach (Draft) February 19, 1996 



Evaluation Overview 

Because details of activities in this particular module are still evolving, aspects of 
the evaluation of this module can be provided only in broad terms. Key evaluation 
questions that will be answered, however, may include: 

• What is the ease of use and effectiveness of the WWW and Lynx “front-end” 
that is developed for accessing Federal information resources? What are the 
trade-offs and benefits/limitations of each type of a front-end? 

• How many and what types of Federal information resources were made 
available to the users? 

• What criteria were used for the selection of Federal information resources to be 
included in this module? 

• How do users assess the appropriateness of these Federal information 
resources? Did project members conduct a needs assessment of users as a 
basis for determining which Federal resources to make available and what 
were the findings from this assessment? 

• To what degree were Federal information resources and services of a local 
nature (providing unique and special interest to the Oregon and Northwest 
community) provided? 

Currently, the GIS component of the project is unclear. Project personnel are in the 
process of describing what this component will be exactly. Evaluation questions and 
criteria will be developed after that component is clarified. 



State Agency Information 

Objectives: 

• Expand public access to basic state government information files. 

• Develop an online state document ordering clearinghouse facility. 

• Provide access to government information from rural libraries via the Internet. 

Additional objectives related to this module that might be considered are: increase 
the public s knowledge of state government activities and services; provide citizens 
with information on how to contact government officials; and promote electronic 
access to and use of state government electronic information services. 
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Evaluation Overview 

In response to the first objective related to expanding public access to basic state 
government, some evaluation questions include: 

• What criteria were used to identify “basic” state government information files? 

• To what degree are there indicators that “public access” was expanded to state 
government information? For example, can it be shown that selective 
populations or target groups have public access as a result of these services? 

• What specific state information resources and services were made available, 
how were they selected, and to what degree do they meet user needs? 

• What specific groups of users found these sources useful and for what reasons 
or purposes? 

Focus group sessions or interviews with selected state government officials may be 
conducted to identify impacts, benefits, and problems that resulted to these agencies 
from being linked electronically to the state’s citizens. 

The online ordering facility can be assessed unobtrusively once it is in place on a 
number of user-based criteria including response time, courteousness of the staff, and 
accuracy of the system; the effectiveness of the process for transferring money 
electronically will be assessed; the "security and verification procedures" will need to 
be clarified so that they can be evaluated. In addition, records will need to be 
maintained as to: 

• The number and type of state government documents that were ordered via 
the system. 

• The transaction response time between the receipt of an order and the delivery 
of the requested publication. 

A sample of those individuals who ordered state publications can be surveyed to 
determine their degree of satisfaction with the service. 

Regarding the rural access aspect of this module, a definition as to what 
constitutes a “rural” community will need to be developed. A typical definition is an 
unincorporated location or a community with population of 25,000 or less and not in a 
SMA. Some more specific assessment questions here are: 

• How many rural communities were connected to the project at what cost? 
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• How many “rural participants” used the online connectivity to access any 
component of the project and what components did they connect to? 

• How did “rural participants” assess the usefulness of the resources made 
available to them? 



For this particular module, it may be useful to conduct a number of site visits to rural 
locations to determine, first hand, how participants are using their connectivity and 
determining the impacts from such connectivity. 



Local Community and Government Information 

Objectives 

• Provide display and delivery of social government services including consumer 
health information, full text of agency documents (information, meeting 
notices, etc.), and calendars. 

• Provide searchable access to local government databases included local 
government service directly and local government personnel phone directory. 

• Provide a gateway to local governmfjntfonline services including metro bulletin 
board, Portland visitors bureau homepage, email to county employees, a 
gateway to each county agency for public interaction, computer conferencing, 
and distance leaming/training in the use of the system. 

• Provide online forms access including park reservations and licenses. 
Evaluation Overview 

This module also will require more detail as to activities and tasking before the 
evaluation component can be finalized. A detail of the existing system configuration 
and the configuration finally developed will depict the changes made (to be done by 
Portals Sysop). Key evaluation questions to be asked, however, include: 

• What and how many social government services and forms were made 
available? 

• How many and what types of users accessed which government services and 
forms? 

• What was the ease with which the services could be accessed by users and to 
what degree did users find these services appropriate and useful? 
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• How users accessed local information via the Portals gateway 

(KlTNet?)? tyhat was the ease of use of the gateway and to what degree did 
■ isers -finthtliis gateway to be appropriate and to help them access the desired 
information? 

• To what degree did the community or governmental agencies find it useful to 
have their agency services or resources available online? What impacts did 
electronic access have their agencies? Do they think they improved the quality 
of services or reduced costs? 

Throughout this module indicators of the degree to which project staff were able to 
obtain coordination and support from various community and governmental agencies 
would be useful. Interviews or focus group sessions with a sample of community 
groups and agency officials that participated in the project will be conducted. 
Interviews or focus group sessions with users of this component of the system can 
identify system strengths and weaknesses. 



State and Local Historical/Archival Information 

Objectives 

• Scan and make available 10,000 maps, original manuscripts and other 
historical documents and make then available over the Internet. 

• Catalog and scan from OHS map collections approximately 500 maps of 
historical significance. 

• Scan and make available photographs documenting the Oregon Country from 
the earliest to the present. 

• Make available online the catalogs and finding aids to its collections of books, 
serials, and oral histories. 

• Create, scan, and make available images of a substantial number of artifacts 
from its material cultural collections. 

• Produce a series of electronic educational packages for Internet access by 
students in grades 4-14 (this is the responsibility of the state library) 

The actual number of items to be scanned and digitized may vary from that proposed 
depending on cost figures — which are being obtained. Thus, targets for the above 
objectives may need to be revised. 
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Evaluation Overview 

• How many and what types of information resources were digitized and made 
available? 

• What was the quality of the images as they appeared on the screen under 
differing levels of connectivity and equipment support? 

• How many and what types of users accessed which of the digitized information 
resources? 

• To what degree were appropriate information retrieval standards used to 
digitize and retrieve the images? Are these standards compatible with other 
imaging storage and retrieval systems? 

• What is the effectiveness of the catalogs and other finding aids that were 
created to access this digitized images and other resources? 

• To what degree are the images and resources that were made available 
appropriate for users or otherwise found to be useful in their daily tasks? 

The last objective in this module is the responsibility of the state Iforngy The 
developers of these educational packages need to develop objectives for the modules 
so that the evaluation team can determine the degree to which the objectives are 
accomplished. In addition, key evaluation questions will be: 

• How many and what types of educational packages were developed? 

• How many and what types of people used these packages? 

• Assuming the packages might be available both in print and online, one can 
compare the use of each type and the degree to which each were found to be 
useful by participants. 

• To what degree were these instructional packages integrated and used by 
teachers in the various schools? How did the teachers assess the usefulness 
and value of the instructional packages? 



User Support 

Portals will develop a "single point of presence" in terms of providing help services 
for using the various information services and resources to be developed for this 
project. 
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Objectives 

• Provide user support and training including an online “helpline.” 

• Emphasize user friendly menus in public interfaces. 

• Conduct user education and staff development programs. 
Evaluation Overview 



The "Helpline" has yet to be designed, specific educational objectives have not be 
detailed, and evaluation cannot proceed until more specifics are known about it. 
Likely evaluation questions to be posed, however, include the following: 

• How many users have asked what types of questions to the helpline over the 
course of the project? 

• To what degree did users obtain accurate, timely, and useful answers to their 
questions? 



• How many and what types of training sessions were conducted to assist users 
in using the system? 

• To what degree did participants at training sessions find the instruction 
appropriate, helpful, and usefiil? 



• How many and what types of online help menus are there on the various 
services? 

• To what degree did users find these help screens and menus appropriate 
helpful, and useful? 



• To what degree were training and educational instruction objectives 
accomplished? 

Some aspects of assessing the helpline lend themselves to unobtrusive testing to 
determine the degree to which the service is accurate, timely, helpful, and the staff 
are courteous. Transaction logs of a sample of the actual questions posed to the 
helpline and the staffs answers can be analyzed by the evaluation study team. 
Questionnaires to workshop participants can provide an assessment of the quality of 
workshops and other types of on-site training. “Pop-up” questionnaires on the help 
screens can also provide information as the quality and usefulness of these screens. 
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Data Collection Methods 

A number of data collection techniques will be used to answer the evaluation 
questions outlined above. These can be used across the various modules: 

• User- and/or surrogate-based tests . Either obtrusive or unobtrusive tests can 
be done on site or by off site surrogates to assess various aspects of the 
information services being provided. This can include not only the information 
itself, but the way it is presented, organized, and linked to other information 
services. 

• User assessments . A key component of the modules is to determine the degree 
to which the services met user needs, the degree to which the user found the 
services useful, and the degree to which the services had “impacts” on their 
daily activities. Focus groups, surveys, and online pop-up questionnaires can 
be used in this area. 

• Anecdotal information and testimonials . There is a use for this type of 
information and we will devise means to track and record such once the 
services and resources start to be made available. Such data can be collected 
from focus group sessions, online suggestion and comment screens, and group/ 
individual interviews. 

• Transaction logs . For each of the various modules the Portals Sysop will need 
to maintain transaction logs that minimally track the number of times a 
particular service or resource is "hit" or used; an analysis of the types of users 
hitting the resource [to the degree possible]; amount of time spent on a 
particular service; patterns of where users move through the services going 
from one to another, etc. 

• Online feedback . Either on a module by module basis or as an overall approach 
in the Helpline, data will be collected describing user suggestions and 
comments for improving the services. The Portals Sysop will be responsible 
for this evaluation component. 

• Site visits . For some of the modules, it may be useful to visit specific schools, 
universities, libraries, or other sites that have been actively involved in using 
the system. Focus groups, individual/group interviews and other data 
collection techniques can be incorporated into the site visit. 

• Pon-up questionnaires . One technique to assess particular electronic services 
are “pop-up” screens that appear on particular Web pages or online services. 
These “pop-up” screens must be short (no more than three questions) and 
quick. Typically they ask the user to assess this particular page or service. 
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Once the user has answered or decided not to answer, the screen will not 
appear again for that particular user. 

• Surveys, focus groups, and interviews . Depending on the particular service to 
be assessed, a print survey, a focus group of key stakeholders, or 
group/individual interviews can be conducted. Surveys can have low response 
rates. Focus groups offer a number of benefits as to being able to probe and 
obtain input from different groups but can require significant preparation time. 

This list is not intended to be comprehensive. Rather, it is suggestive of the specific 
types of data collection that can be employed. Decisions about which types of data 
collection are best for specific aspects of the modules will be determined once more 
information about those modules is available. 



Schedule 

Monthly reports of time commitment by project personnel began January, 1996. 
The transaction logs for various services and resources is being implemented and will 
need to be developed for each service as that service becomes available. 

The scheduling and development of individual data collection instruments will be 
done after the project evaluator receives additional detail on- the components of each 
of the modules, additional refinement of this document is accomplished, and the 
evaluation research assistant has been hired. Eventually there will be a timeline and 
tasking of which data collection activities will occur when. 



TO: 



ix P-5" 



Howard McGinn 





January 30, 1997 



FROM: Chuck McClure 

RE: Summary of January 22-23, 1997 Meetings in Portland 

During January 22-23, 1997 McClure traveled to Portland and met with the 
Portals staff (Howard, Dennis, and Karen), staff at the Oregon Historical Society 
(OHS), a representative from the State Library (Ernest); and staff from the 
Multnomah County Library regarding the various modules contained in the 
Department of Education (DoE) grant known as Hatfield I. The purpose of this 
memo is to (1) briefly summarize the meetings and topics discussed, and (2) record 
key decisions made regarding the completion of the project. 



January 22: Meeting with OHS 

Howard and Chuck spent the morning of January 22 at the OHS. We began by 
meeting with OHS staff who have been involved on the project — ten people, the lead 
person being Sue Seyl. Chuck opened the meeting by providing an overview of the 
current project status. The major points Chuck stressed at the meeting were: 

• Evaluation Research Assistant . Despite a major effort on the part of Howard 
and Chuck during the past six months, we had not been able to hire an 
evaluation research assistant locally. Thus, no on-site data collection 
activities such as focus groups, etc. had been done to date or would be done in 
the foreseeable future. 

• Emphasis on Self-assessment and Reporting . Due to the inability to hire a 
research assistant, the evaluation strategy would shift to emphasize more 
self-assessment and reporting on project activities by the key participants in 
the project, i.e., Portals, OHS, State Library, and Multnomah County Library. 
In effect, the originally proposed evaluation (dated February, 1996) would be 
modified considerably. 

• Changes in Project Obiectives/Activities . For a host of reasons, some of the 
proposed project objectives/activities would need to be eliminated or modified. 
For example, new technologies and knowledge that was not present when the 
project was proposed affected what needs to be done or what can be done. It 
was important to identify those NOW and explain the changes in the status 
report (see below) and to report these to DoE. 

• Responsibilities for Self-assessment . OHS (with Sue taking the lead) would be 
responsible for developing a 5 page status report (with appropriate and 
numerous appendices) describing: 
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project activities and accomplishments to date — especially in terms of 
accomplishing the original objectives as outlined in the proposal and 
noting changes to the original objectives. 

current status of work on the project. 

next steps to be taken. 

- assessment of the project in terms of extensiveness, efficiency, 
effectiveness, and impact; especially important is to include statistics of 
work done - counts of items disseminated, items cataloged, items made 
available on the web, time committed to the project by staff, costs, etc. ; 
testimonials and anecdotes or success stories; presentations; in-kind 
contributions from OHS, and activities related to dissemination of 
findings or products or information related to the project. 

lessons learned, thus far, from the project -- lessons that might help 
others who are considering doing similar activities. 

Chuck asked that he receive these status reports by Friday, February 7, 1997. 

Sue, et. al. then toured Howard, Karen, and Chuck through OHS work areas where 
we reviewed the OHS Website and the content that is being developed for that site. 
We discussed a range of topics and issues related to their work on putting maps, 
images, photographs, and other information up on their Website. Overall, OHS is 
making excellent progress on developing this site and moving print based information 
into digital format. 

We also reviewed the various work forms, data entry forms, and other 
code/software that Richard and Todd developed. Chuck recommended that a product 
that should be made available to others from their work was a CD that had all these 
forms and software so that others digitizing maps, photographs, etc., would not have 
to reinvent the wheel. Howard agreed to work with Sue on obtaining support either 
through this grant or a later Hatfield grant to purchase a CD presser so that such 
could be done. 

We next visited the photolab where they demonstrated the use of a magnetic 
board as part of the process for digitizing photographs for the Website. They also 
described the off-site facility used for larger photographs and maps. 

The visit concluded by meeting with Chet Orloff (attended by Sue, Karen, Chuck, 
and Howard). Chuck told Chet that OHS had been making good progress on project 
activities. We also discussed additional possible funding sources for project activities. 
Chuck identified and described a number of possible programs and offered to provide 
Chet with electronic introductions or follow-up information to folks at these sources 
[these have since been done by Chuck]. 
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Overall, we came away from this site visit to OHS with a very positive 
perspective regarding this module. OHS staff have been making excellent progress in 
digitizing a range of archival information to be posted on their Website; their Website 
is beginning to take form and is under development with the likelihood of being made 
public in the near future; they have a very enthusiastic attitude toward the project 
and are committed and very interested in their work; the administrator of OHS is 
very supportive and engaged in the project; and they are learning a great deal about 
developing a Website with a range of digitized information resources. 



January 22: Afternoon Sessions at Portals 

We touched on a number of issues and topics during the afternoon of January 22 
at Portals with Howard, Karen, and Dennis. The most significant topics and decisions 
were the following: 

• Federal Information Component . The federal information component as 
outlined in the original grant has been largely superseded by events. A number 
of excellent Federal Web locators are up and available on the net and need not 
be duplicated by the project. Thus, the homepage for the federal information 
resources on the Portals page should be a pointer to those locators and selected 
other key sources with annotations. 

Purchase of discreet federal databases or licensing of those databases (such as 
Stat-USA) would be continued rather than purchase and mounting of 
government CDs — many of which are available via federal websites. 

The homepage for the federal information on Portals should be re-worked to 
provide links to federal locators, made a bit more user-friendly, and link to other 
federal collections or note appropriate federal information at Portals member 
libraries. 

In terms of a specific training component for this aspect of the project we 
discussed the possibility of some Web-based instructional guides or directions 
to some such guides on the net being provided on the homepage. Given the 
evolution of the Federal Webpage component of the project, Howard believed it 
to be unlikely that “training modules” would be developed for use of Federal 
information on Websites. 

Regarding administration of this component, Karen should become more 
directly involved in revised the homepage and making links. Since a number of 
aspects of this module have been superseded by events, it is unclear if the 
originally proposed advisory committee, etc., is needed except for informal 
feedback and comments. 
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GIS Component . Apparently, the GIS component described in the original 
proposal was never intended for inclusion. The purpose of this component vis a 
vis other project activities is unclear. At the time of proposal submission, this 
component was based on a project then in process which may or may not have 
come to fruition. Howard will check with others at PSU about this proposed 
component and indicate in the letter to the DoE the changes to be made. 

User Services. Training, and User Feedback . There are a number of instances 
in the original proposal that refer to user-based feedback, training, etc. 
Specific attention will be given by Karen to redesigning the Federal Web 
portion of the Portals Website to e more “user friendly.” This may also include 
some structured techniques to obtain user assessment of the Portals Website 
and more particularly the Federal homepage. 

In terms of developing training and user support services. We discussed the 
possibility of establishing a “virtual help desk” where users could leave 
questions or ask reference questions electronically to a network of reference 
librarians within the Portals membership. Karen will explore this possibility in 
greater detail. 

Evaluation Component . Chuck proposed a number of options to Howard 
regarding strategies to evaluate the project given the fact that we had not been 
able to hire anyone as the Evaluation Research Assistant. After considerable 
discussion and weighing the trade-offs, we decided to: 

not continue looking for an on-site assistant given the results of past efforts 
and because the start-up time and level of effort to get someone up to speed 
would be excessive at this point in the project. 

propose to Joe Ryan, a doctoral student at Syracuse University who has 
worked with Chuck on a number of similar projects — and worked with 
Howard when he was state librarian in North Carolina — to conduct an on- 
site evaluation later in the project. 

Chuck would propose to Joe to come to Portland and conduct site visits, 
focus groups, and other data collection activities the end of June, 1997. 
This 10 day visit would serve as the major evaluation effort. He would also 
commit another 10 days of consulting in preparation for the visit and in 
writing up the report of the visit. The 20 days consulting at $300 per day 
plus a maximum of $2000 for travel would be a $10,000 expenditure instead 
of that originally budgeted for the evaluation assistant. Chuck will discuss 
this proposal with Joe upon his return to Syracuse. Any additional 
negotiation regarding fees and time would be done directly between Joe and 
Howard. 



- increase Chuck’s commitment to the project during the Months of February 
- September to continue ongoing evaluation consulting, help in the design of 
the June assessment and final report writing which would have been done 
by the research assistant but will now be done by Joe. Chuck estimates 
that he will need a total of 14 days for February - September, assuming Joe 
is hired and assuming two more trips for Chuck to Portland (three trips are 
currently budgeted). 

In fact, these changes in the evaluation approach will result in less total 
expenditures within the evaluation component and still result in adequate 
evaluation information and reporting for the project. Howard will include these 
changes in his report to DoE as part of overall changes in project activities and 
objectives. 



January 23: Morning Session 

During the morning of January 23 we conducted a meeting at PSU attended by a 
representative of the state library, Ernest Perez, and three staff from OHS, Sue, 
Richard, and Todd. Dennis, Karen, and Howard also attended the meeting. This 
meeting had a number of objectives which will be discussed below. 

A first objective was for Chuck to brief Ernest on the current status of the project 
and the need for the state library to develop a status report. Basically, Chuck 
reviewed the information that he discussed the previous morning at OHS - which is 
summarized on page 1 of this report and will not be repeated here. Ernest agreed to 
develop the status report as requested. 

Secondly, Ernest provided a status report on the degree to which the various 
objectives for the State Library were being accomplished. 

• The state library information system was under development, information was 
being loaded and made available, some problems with Harvest had delayed the 
search engine, and additional sources would be mounted. 

• The sun workstation was operable and would soon be put to use on a range of 
activities related to the project. 

• For the rural public library project 25 libraries had received equipment and 
were now connected to the Internet and were provided services. Training had 
been done by the State Library. 

• Centralized access to and ordering/purchase of those publications 
electronically was under development. The state printer runs this server, a 
planning and marketing effort was underway as well. 
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We briefly discussed the key issues related to completing these projects. Overall, the 
State Library has been doing a good job on these projects and especially with the 
deployment of the computing equipment and connections to the rural libraries. 

Third, Chuck provided a brief tutorial to meeting attendees on the status of the log 
analysis that we would be using on the Portals, OHS, and State Library Websites. 
He reviewed the various types of logs that could be used for analysis, how the log 
statistics would be used for this project, and how the log statistics could be used by 
Portals, OHS, and State Library for planning and Website development. We expect 
to analyze Website log files during a specific test period later in the project at all three 
sites. 

We concluded this section with a demonstration by Dennis of various Websites 
that provided log analysis software and recommended two specific packages that 
OHS and the state library should examine further to be mounted on their site. The 
State Library and OHS understood that Dennis would provide moral support but 
could not actually install this software. 



January 23: Lunch with Multnomah County Library Representatives 

Howard and Chuck met with Donna Reed and Jeanne Goodrich over lunch to 
discuss their development of the community information network and issues related 
to the tatu of this component of the project. Donna and Jeanne brief Howard and 
Chuck of the impact of Proposition 47 on the library in terms of significant budget 
cut, staff lay-offs, resource reallocations, and other fall-out. 

Chuck briefed Donna and Jeanne about project status and the need for a written 
status report. Again, the information presented on page 1 of this report was also 
presented to Donna and Jeanne. Donna agree to develop the report as requested. 
Donna summarized the status of the development of the community information 
network, RIT Net. Basically they have the network up and running and a significant 
amount of very useful content and links. 

After lunch we went over to the administrative offices of the library and received a 
demonstration and tutorial of RIT Net. Details on the content and organization of 
RIT Net will not be provided here. Those interested may wish to go directly to the 
Website at <http://www.multnomah.lib.or.us/>. Chuck offered a number of 
suggestions for content and organization regarding the Website. Suffice to say that 
they have made good progress on this module of the project. 

During the discussions regarding Multnomah County Library’s involvement in the 
project it became obvious that they can contributed much to the development of RIT 
Net beyond that originally proposed. Further, Donna outlined a number of possible 
enhancements, changes, additions, and new content that could be provided to RIT 
Net between now and the end of the project. 





Howard asked Donna to submit to him a proposal with incremental costs and 
enhancements for RIT Net that might be done now as part of the project. Based on 
his review of her proposal, the merits of the enhancements being proposed, and the 
availability of existing resources, or resources that might be reallocated given other 
changes that are being made in the project, resources might be made available to RIT 
Net to implement these enhancements. 

There was agreement between Howard and Chuck (later) that enhancing RIT Net 
activities would be especially appropriate given the spirit of the original proposal and 
the likelihood that some of the original objectives would not be implemented (see 
above). This change should be noted and reported to DoE. 



January 23: Afternoon Session at Portals 

During the afternoon of January 23 McClure met with Howard and also with 
Dennis and covered a number of topics. These included the following: 

• Log Analysis . We discussed at length specific strategies to implement the log 
analysis software on the Portals Website. The existing software is 
cumbersome compared to others available. We agreed that Todd would be 
contacted immediately by Dennis to begin installation of the new log analysis 
software. 

We also discussed in some detail the type of statistics that should be 
represented on the Website, the need for Howard to have composite access 
statistics by Portals institution for specific data bases and other services on 
the Website, and the importance of these statistics both for (1) project usage 
information , and (2) basic management information for Howard to assess 
Portals activities and report to his Board. 

We agreed that Todd would have lead responsibility for this with oversight by 
Dennis and that Todd would start on the log analysis project as soon as 
possible. We expect to have the final version of the Web statistics working on 
the Website and finalized by April 1. We also agreed that we would use an 
iterative process of Howard reviewing the representation and presentation of 
statistics on the Website to produce, eventually, the format and content 
necessary. Chuck will also review the content and presentation of statistics 
with Howard. 

• Statistics on Deployed Workstations . We discussed the forthcoming 
deployment and installment of the 14 workstations to Portals member 
institutions and decided that we would not software or sniffers to keep track of 
traffic, URLs, time in use, or other statistics from that end. Instead, we 
discussed the possibility of tracking the degree to which these 14 workstations 
(through tracking their IP addresses) used the Portals Website. This tracking 



could be done once the Portals log analysis software is installed. Specifics of 
the data we would track from these 14 deployed workstations would be 
discussed at a later date. 



Overview 

The two days of meetings were extremely productive. Considerable time was 
spent on meeting with project participants and reviewing their status regarding 
project objectives. The site visits confirmed that good efforts were being made on 
project components. Effort was also dedicated to revising and restructuring aspects 
of the project in a “mid-course” correction to insure maximum benefit from remaining 
project resources. 

To a large degree, the visit provided formative evaluation from McClure for each of 
the modules. McClure was able to review the status of project activities, confer with 
project participants, and suggest strategies to improve project performance and 
outcomes. Thus, the visit constituted a key component of the overall evaluation 
methodology. 

The one module of the project that required the most attention was the Federal 
government component. A number of changes and revisions in this aspect of the 
project were required for a host of reasons — not the least of which being that a vast 
amount of Federal information is already available and relatively well-organized on 
the net. However, the changes to be made (see above) in this component of the 
project should still assist in making Federal information more accessible to Portals 
member institutions and others using the Portals Website. 

A final important result of the meetings was how to best restructure the 
evaluation component with one on-site visit by a well-trained and knowledgeable 
evaluator rather than relying on the on-site Evaluation Research Assistant who we 
have been unable to hire. While some additional planning and evaluation methods 
development will be need to implement the strategy decided upon, we believe that the 
approach will result in collecting important evaluation information. 
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PORTALS 

Portland Area Library System 
P.O.Box 1151 

Portland, Oregon 97207-1151 



March 25, 1997 



Ms. Christina Dunn 

Director, Discretionary Programs Division 
Library Programs Office 
U.S. Department of Education 
Room 300 

555 New Jersey Avenue, NW 
Washington, DC 20208-5571 

Re: Grant No. R039D50010-95A 



Dear Chris, 

This letter is in reference to Grant No. R039D50010-95A, “Citizen Access to Government and 
Other Information”. I am writing to request that the following changes in the grant be noted and 
authorized by your office. The changes listed below were compiled with the assistance of Dr. 
Charles McClure. Dr. McClure has been engaged by Portland State University as the Chief 
Evaluator for the grant. 

Federal Information Module 

NOTE: Charles McClure, in the report of his evaluation visit to PORTALS, January 22 - 23, 
1997, noted: 

“The federal information component as outlined in the original grant has been largely superseded 
by events. A number of excellent Federal Web locators are up and available on the net and need 
not be duplicated by the project. Thus, the homepage for the federal information resources of the 
PORTALS page should point to those locators and selected other key sources with annotations. 

Purchase of discreet federal databases or licensing of those databases (such as Stat-USA) would 
be continued rather than purchase and mounting of government CDS -- many of which are 
available via federal websites. 

The homepage for federal information on PORTALS should be re-worked to provide links to 
federal locators, made a bit more user-friendly, and link to other federal collections or note 
appropriate federal information at PORTALS libraries.” 

Based on the above, the following changes would be made: 
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Page 2. 

Change permission request: 



I. O riginal : A librarian with documents experience and excellent technical skills will be hired to 
develop a World Wide Web page (graphical) and Lynx (ASCII) structure for federal, and other 
government information. 



Change: A full time documents librarian would not be hired. 



Budget effect Decrease in salary line item of $71,050; decrease in benefits line item of 

$26 289. Funds will be reprogrammed at a later date. I will notify you about the use of these 
funds as soon as possible. 



Z Or iginal: PORTALS would mount, on-line, “Census data, National Trade Data Bank, 
National Environmental, Sociological Data Bank, Regional Economic Information System ” The 
files would be stored in CD-ROM format. 

£h an g e: Databases would not be mounted online because this work would be duplicative. Web 
site development of links to major federal information web sites and the purchase of access to 
appropriate databases (e.g. StatUSA) will replace original plans. 

Budget effect: None. 

— Original: PORTALS will deploy a geographical information system to present data in user 
oriented graphical format. 



Change. Development and implementation of a GIS system will not take place. The GIS project 
was included in the original grant abstract. When the budget revisions were made at the time of 
the original award, all funding for this component was deleted. However, the GIS project 
description was not removed from the abstract when the budget revision was submitted. There is 
no money in the grant to implement this component. 

Budget effect : None. 

Local Government Information Module 

Change: The Multnomah County Library (MCL) has developed a community information web 
service called “RITNET”. But as noted by Dr. McClure in his onsite visit report, “it became 
obvious that the library can contribute much to the development of RITNET beyond that 
originally proposed”. There was agreement between Dr. McClure and me that enhancing RITNET 
activities would be especially appropriate given the spirit of the original proposal. I request 
permission to transfer funds to MCL to carry out these plans. 
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Budget transfer request: Request permission to transfer funds from the salary budget line item to 
an increase in the sub-contract with MCL of $50,000. A specific budget will be submitted if you 
approve this transfer. 



Information System Module - User Support 

Original: In his report Dr. McClure noted: “There are a number of instances in the original 
proposal that refer to user-based feedback, training, etc.” 

Change: Because of new technology, the following user services programs will be investigated 
and/or developed. Changes are based on recommendations made by Dr. McClure. 

Specific attention will be given by PORTALS staff to redesigning the Federal Web portion of the 
PORTALS website to be more ‘user friendly’. This may also include some structured techniques 
to obtain user assessment of the PORTALS website and more particularly the Federal homepage. 

In terms of developing training and user support services, a “virtual help desk” where users could 
leave questions or ask reference questions electronically to a network of reference librarians 
within the PORTALS membership will be developed. 

Budget effect: None. 



Evaluation Module 

Original. An Evaluation Research Assistant will be hired on a .5 FTE basis. This person would be 
responsible for data collection that would be used to evaluate the project. This person’s work 
would be overseen by Dr. McClure. The grant also created an implementation advisory team. 

Change: An Evaluation Research Assistant will not be hired. The implementation advisory 
committee has not been convened to date. 

As outlined by Dr. McClure, the evaluation will be conducted as follows: 

1 . Self-Assessment and Reporting. Key participants in the project would conduct a self- 
assessment and report findings on a regular basis to Dr. McClure. 
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2. Joseph Ryan, a doctoral student at Syracuse University who has worked on a number of similar 
projects with Dr. McClure (and with me in North Carolina), will be hired to conduct a ten day 
concentrated evaluation of the project in June 1997. Mr. Ryan would also commit an additional 
10 days of consulting to the project for data interpretation and report writing in Syracuse. 

3. Dr. McClure would spend up to fifteen additional days on the evaluation process between 
February and September 1997. 

R eason for change: PORTALS and Portland State University conducted regional searches for an 
Evaluation Research Assistant. The searches were not successful. Thus no on-site data collection 
such as focus groups, etc. have been done to date or would be done in the foreseeable future. 

Budget effect: Decrease in personal services (other) line item of $7,100. 

While the above are programmatic changes, they may or may not affect the budget. At this point I 
am primarily interested in our receiving clearance to move ahead with programmatic changes. Any 
changes in budget will be submitted at a later date pending further review. 



Please let me know if you have questions. 



Sincerely, 






Howard F: McGinn 
Executive Director 



cc: Dr. Charles McClure 

Ms. Judy Ngai, Sponsored Projects Administrator, Portland State University. 
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UNITEO STATES DEPARTMENT OF EDUCATION 

OFFICE OF THE ASSISTANT SECRETARY 
FOR EDUCATIONAL RESEARCH AND IMPROVEMENT 



MAR 27 1997 

Dear Howard: 



J *™ ived y° ur letter dated March 25, 1997 concerning grant # 
R039D50010-95A, "Citizen Access to Government and Other 
Information." 



I concur with Chuck McClure's findings regarding the federal 
information component and the RITNET contribution to the project. 

As you will not be developing a WWW page and Lynx structure for 
federai and other government information, you do not have to hire 
the documents librarian as called for in the original proposal 
Funds budgeted for this purpose ($71,050 from the salary line 
item and $26,289 from the benefits line item) should be moved to 
other budget lines. 

I understand that mounting certain online databases (Census Data, 
National Trade Data Bank, National Environments, Sociological 
Data-Bank, Regional Economic Information System) would be 
duplicative. It is appropriate to replace this approach by 
providing Web site development of links to major federal 
information Web sites and by purchasing access to appropriate 
databases. While this may not change budget expenditures, if it 
causes changes in line items, you will need to clear that with 
this office. 

I have noted for the record that the GIS project which was 
included in the original grant proposal, was deleted in the 
original budget revisions. I understand that funds are no longer 
earmarked to implement this component. 

You may use grant funds to support the Multnomah County Library's 
RITNET; the $50,000 cited seems reasonable for this activity 
Since you are reducing expenses under the Salaries line item,’ it 
seems reasonable to use such funds for this activity. As this 
will probably result in changes in line items, you will need to 
clear those changes with this office. 

Also, it makes good sense to redesign the Federal Web portion of 
the PORTALS Website to make it more user friendly. The virtual 
help desk sounds like a good approach. Since this will not 
affect the budget, you may undertake this activity. 

Changing the evaluation strategy as Chuck McClure suggested 
should not change the quality or content of the evaluation. You 
may make the . change as requested. I understand that the change 
will result in a decrease in the personal services (other) line 
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item by $7,100. 

Szrs «v cceptaw 

revised budget demonstrating the changes P ' send me a 



Sincerely, 




Chris Dunn 
Director, Discretionary Program 
Library Programs Office 
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March 25, 1997 



Ms. Christina Dunn 

Director, Discretionary Programs Division 
Library Programs Office 
U.S. Department of Education 
Room 300 

555 New Jersey Avenue, NW 
Washington, DC 20208-5571 

Re: Grant No. R039D50010-95A 

Dear Chris, 

This letter is in reference to Grant No. R039D50010-95A, “Citizen Access to Government and 
Other Information”. I am writing to request that the following changes in the grant be noted and 
authorized by your office. The changes listed below were compiled with the assistance of Dr. 
Charles McClure. Dr. McClure has been engaged by Portland State University as the Chief 
Evaluator for the grant. 

Federal Information Module 



NOTE: Charles McClure, in the report of his evaluation visit to PORTALS January 22 - 23 
1997, noted: 

“The federal information component as outlined in the original grant has been largely superseded 
by events. A number of excellent Federal Web locators are up and available on the net and need 
not be duplicated by the project. Thus, the homepage for the federal information resources of the 
PORTALS page should point to those locators and selected other key sources with annotations. 

Purchase of discreet federal databases or licensing of those databases (such as Stat-USA) would 
be continued rather than purchase and mounting of government CDS - many of which are 
available via federal websites. 

The homepage for federal information on PORTALS should be re-worked to provide links to 
federal locators, made a bit more user-friendly, and link to other federal collections or note 
appropriate federal information at PORTALS libraries.” 

Based on the above, the following changes would be made: 
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Change permission request: 

1. Original: A librarian with documents experience and excellent technical skills will be hired to 
develop a World Wide Web page (graphical) and Lynx (ASCII) structure for federal, and other 
government information. 

Change: A full time documents librarian would not be hired. 

Budget effect: Decrease in salary line item of $71,050; decrease in benefits line item of 
$26,289. Funds will be reprogrammed at a later date. I will notify you about the use of these 
funds as soon as possible. 



T Original: PORTALS would mount, on-line, " Census data, National Trade Data Bank, 
National Environmental, Sociological Data Bank, Regional Economic Information System. ” The 
files would be stored in CD-ROM format. 

Change: Databases would not be mounted online because this work would be duplicative. Web 
site development of links to major federal information web sites and the purchase of access to 
appropriate databases (e.g. StatUSA) will replace original plans. 

Budget effect: None. 

L Original: PORTALS will deploy a geographical information system to present data in user 
oriented graphical format. 

Change: Development and implementation of a GIS system will not take place. The GIS project 
was included in the original grant abstract. When the budget revisions were made at the time of 
the original award, all funding for this component was deleted. However, the GIS project 
description was not removed from the abstract when the budget revision was submitted. There is 
no money in the grant to implement this component. 

Budget effect : None. 

Local Government Information Module 

Change: The Multnomah County Library (MCL) has developed a community information web 
service called “RITNET”. But as noted by Dr. McClure in his onsite visit report, “it became 
obvious that the library can contribute much to the development of RITNET beyond that 
originally proposed”. There was agreement between Dr. McClure and me that enhancing RITNET 
activities would be especially appropriate given the spirit of the original proposal. I request 
permission to transfer funds to MCL to carry out these plans. 
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Budget transfer request: Request permission to transfer funds from the salary budget line item to 
an increase in the sub-contract with MCL of $50,000. A specific budget will be submitted if you 
approve this transfer. 



Information System Module - User Support 

Original. In his report Dr. McClure noted: “There are a number of instances in the original 
proposal that refer to user-based feedback, training, etc.” 

Change. Because of new technology, the following user services programs will be investigated 
and/or developed. Changes are based on recommendations made by Dr. McClure. 

Specific attention will be given by PORTALS staff to redesigning the Federal Web portion of the 
PORTALS website to be more ‘user friendly’. This may also include some structured techniques 
to obtain user assessment of the PORTALS website and more particularly the Federal homepage. 

In terms of developing training and user support services, a “virtual help desk” where users could 
leave questions or ask reference questions electronically to a network of reference librarians 
within the PORTALS membership will be developed. 

Budget effect: None. 



Evaluation Module 

Original: An Evaluation Research Assistant will be hired on a .5 FTE basis. This person would be 
responsible for data collection that would be used to evaluate the project. This person’s work 
would be overseen by Dr. McClure. The grant also created an implementation advisory team. 

Change. An Evaluation Research Assistant will not be hired. The implementation advisory 
committee has not been convened to date. 

As outlined by Dr. McClure, the evaluation will be conducted as follows: 

1 . Self-Assessment and Reporting. Key participants in the project would conduct a self- 
assessment and report findings on a regular basis to Dr. McClure. 
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2. Joseph Ryan, a doctoral student at Syracuse University who has worked on a number of similar 
projects with Dr. McClure (and with me in North Carolina), will be hired to conduct a ten day 
concentrated evaluation of the project in June 1997. Mr. Ryan would also commit an additional 
10 days of consulting to the project for data interpretation and report writing in Syracuse. 

3. Dr. McClure would spend up to fifteen additional days on the evaluation process between 
February and September 1997. 

Reason for change: PORTALS and Portland State University conducted regional searches for an 
Evaluation Research Assistant. The searches were not successful. Thus no on-site data collection 
such as focus groups, etc. have been done to date or would be done in the foreseeable future. 

Budget effect: Decrease in personal services (other) line item of $7,100. 

While the above are programmatic changes, they may or may not affect the budget. At this point I 
am primarily interested in our receiving clearance to move ahead with programmatic changes. Any 
changes in budget will be submitted at a later date pending further review. 



Please let me know if you have questions. 
Sincerely, 



Howard F. McGinn 
Executive Director 

cc: Dr. Charles McClure 

Ms. Judy Ngai, Sponsored Projects Administrator, Portland State University. 
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Instructions 

Jumpstart Grant Application 



Line-by-line Instructions 



1. Applicant 

Enter the full legal name of the applicant library, system or other agency. 

2. Address 

Enter the mailing address, city and zip code of the applicant. 

3. Contact 

Enter the name of the person responsible for the application who may be contacted for further 
information or clarification. This person need not De the intended project manager or the library 
director. 

4. Project manager 

A project manager must be designated at the time the application is submitted. The project manager is 
the person who will be responsible for seeing that the objectives of the project are met. Project 
managers must sign the certification form (Appendix B) to indicate their acceptance of the 
responsibilities orjumpstart project managers. 

5. Fiscal agent 

If the fiscal agent for the project is different from the applicant, enter the name, address, contact 
person, and phone of this agency. An official of the fiscal agent must certify the willingness of their 
organization to adhere to the conditions of the grant (see fflO below). 

6. Target area 

Indicate the geographic area to be served by the project. This will likely be one or more cities, 
counties or portions of counties. Use the names of the cities and/or counties to describe the project 
service area (Example: Tolk, Yamhill, and Marion Counties"; "Newport and surrounding areas in 
Lincoln County"; "La Grande, Pendleton, Hermiston and Baker") 

7. Estimated number of persons served 

Estimate the number of persons that will actually be served by the project. Do not use the total 
estimated population of the service area, but only the persons that you estimate will be directly 
served by the project. 

8. Assessment of needs and objectives of the grant project 

Briefly discuss why a Jumpstart grant is needed to provide the library with Internet connectivity. 
Describe the needs that library customers have for access to the Internet and how the grant might 
provide broader benefits to the community. Then describe how the library will use the equipment 
and Internet service provided by the grant. What sorts of training and orientation sessions will be 
offered for potential users? Where will the equipment be located in the library and how will its 
availability oe publicized in the community? What objectives will the library set for the 12 months 
of the grant project? 

9. Continuation of the project 

Briefly describe how the library intends to continue with Internet services beyond the 12 month 
period of the grant. How will the cost of the Internet connection be by borne by the library? How 
will maintenance of hardware and software be provided for. Will the library continue to offer 
training and support for library customers in using the service? 

10. Certification of the fiscal agent 

An individual vested with the authority to enter into contracts for the fiscal agent (e. g., city, county, 
school district) must certify his/her knowledge of the application, the validity of its contents, and 
understanding of the terms of the grant award 

PLEASE CALL BOBBY ARVIZU AT THE STATE LIBRARY (378-2112, Ext. 254) IF YOU 
HAVE QUESTIONS OR NEED CLARIFICATION OF THESE INSTRUCTIONS. 




Oregon State Library 
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OREGON STATE LIBRARY 



Date Received 

Application Number 

Appendix A 



Jumpstart Grant Application 

(This form may be replicated on a microcomputer, but may not exceed two pages in length) 



1. Applicant: 

2. Address: 

3. Contact person: Phone: 

4. Project manager (if different from contact person): 

Phone: 

5. Fiscal Agent (if different from applicant): 

Name and address: 

Contact person: Phone: 

6. Description of target area to be served by the project: 

7. Estimated number of persons served by the project: 

8. Briefly describe, in the space provided, how the Jumpstart grant would assist the library in 
meeting the needs of library customers/community members. What specific 
objectives/activities would be supported by the Jumpstart grant?: 
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OREGON JUMPSTART INTERNET PROJECT 

A partnership activity of the Oregon Information Highway Project 

among 

the Oregon State Library, 

Oregon State University Library, 
and 

PORTALS - Portland Area Library System 




Library Services and Construction Act 
Higher Education Act 



Grant Activities Report 




Monthly Report# (1-12) 



OREGON JUMPSTART LIBRARY: Seaside Public Library 

Grant Project Number: 3-95-4.3 (a) 29 



Date Submitted: Hav 1. 1996 Submitted By: Paula Lee Clark 

1 . Report the number of search sessions conducted. A session is one person 
(staff or public) searching on the Jumpstart workstation between observable 
start and stop times. (You need not report the amount of time in each 
session.) 



155 



2. Report the number of connect hours per month. (This can be obtained from 
your Internet service provider.) 

180 



3. Report the number of presentations to promote the service. 

business^ governments schools 

community groups (Elks, etc) 

4. Report the total audience number at all presentations combined. 30 

(This is to be counted by library or project presenters.) 

5. Report the number of public instruction hours. (This is to be counted by 
library or project trainers.) 

individual 103 group * o 

6. Include copies of press releases issued with this report. 

7. Include copies of media coverage gained. Examples are copies of articles, 
date/time of radio/tv coverage, and newsletters. 

8. Include success stories, particularly for those who obtain information from 
the government databases. Get permission to use the patron’s story, using 
the accompanying form. 

ERIC 17? 




OREGON JUMPSTART INTERNET PROJECT 

A partnership activity of the Oregon Information Highway Project 

among 

the Oregon State Library, 

Oregon State University Library, 
and 

PORTALS - Portland Area Library System 




Library Services and Construction Act 
Higher Education Act 



Grant Activities Report 




onthly Report # (1-12) 



OREGON JUMPSTART LIBRARY: Seaside Public Library 

Grant Project Number: 3-95-4.3 (a) 29 

Date Submitted: June 4 » 1996 Submitted By: 



Paula Lee Clark 



1. Report the number of search sessions conducted. A session is one person 
(staff or public) searching on the Jumpstart workstation between observable 
start ana stop times. (You need not report the amount of time in each 
session.) 



101 



2. Report the number of connect hours per month. (This can be obtained from 
your internet service provider.) 

133 



3. Report the number of presentations to promote the service. 

business government school 1 Mldd1e scho ol 

community groups (Elks, etc) Radi -°. : KSWB Ma ^ 14 * 1996 

4. Report the total audience number at all presentations combined. i_ii_£ 

(This is to be counted by library or project presenters.) 

5. Report the number of public instruction hours. (This is to be counted by 
library or project trainers.) 

individual so groups £ 

6. Include copies of press releases issued with this report. 

7. Include copies of media coverage gained. Examples are copies of articles, 
date/time of radio/tv coverage, and newsletters. 

8. Include success stories, particularly for those who obtain information from 
the government databases. Get permission to use the patron's story, using 
the accompanying form. 

o 

ERIC 
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OREGON JUMPSTART INTERNET PROJECT 

A partnership activity of the Oregon Information Highway Project 

among 

the Oregon State Library, 

Oregon State University Library, 
and 

PORTALS - Portland Area Library System 



Library Services and Construction Act 
Higher Education Act 



Grant Activities Report 

OREGON JUMPSTART LIBRARY: 

Grant Project Number: 3 - 1 S - M '5 fo-'b ^ R 




onthly Report #(1-12) 






Date Submitted: V(» - ^ Submitted By: u, . . . r ^ „ d.c.r. 

1. Report the number of search sessions conducted. A session is one person 
(staff or public) searching on the Jumpstart workstation between observable 
start ana stop times. (You need not report the amount of time in each 
session.) 



'n 



2. Report the number of connect hours per month. (This can be obtained from 
your Internet service provider.) 

\p q 

3. Report the number of presentations to promote the service. 

business government school i 

community groups (Elks, etc) 

4. Report the total audience number at all presentations combined..^ 

(This is to be counted by library or project presenters.) 

5. Report the number of public instruction hours. (This is to be counted by 
library or project trainers.) 

individual group • 3 

6. Include copies of press releases issued with this report. 

7. Include copies of media coverage gained. Examples are copies of articles, 
date/time of radio/tv coverage, and newsletters. 

8. Include success stories, particularly for those who obtain information from 
the government databases. Get permission to use the patron's story, using 
the accompanying form. 
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OREGON JUMPSTART INTERNET PROJECT 

A partnership activity of the Oregon Information .Highway Project 

among 

the Oregon State Library, 

Oregon State University Library, 
and 

PORTALS - Portland Area Library.System 



Library Services and Construction Act 
Higher Education Act 



Grant Activities Report 




nlhly Reports (1-12) 



OREGON JUMPSTART l IRRARY: Seaside Public Lib rary 

Grant Project Number: 3-95-4.3 (a)2 9 

Date Submitted: 8-3-96 Submitted By: Paula Lee Clark 

1. Report the number of search sessions conducted. A session is one person 
(staff or public) searching on the Jumpstart workstation between observable 
start and stop times. (You need not report the amount of time in each 
session.) 

73 



2. Report the number of connect hours per month. (This can be obtained from 

your Internet service provider.) 

120 



3. Report the number of presentations to promote the service. 

business government__ school 

community groups (Elks, etc) 

4 Report the total audience number at all presentations combined. 

(This is to be counted by library or project presenters.) 

5. Report the number of public instruction hours. (This is to be counted by 
library or project trainers.) 

individual 10 group 

6. Include copies of press releases issued with this report. 

7. Include copies of media coverage gained. Examples are copies of articles, 
date/time of radio/tv coverage, and newsletters. 

f • • •• 

8. Include sufccess stories, particularly for those who obtain information from 
the government databases. Get permission to use the patron s story, using 
the accompanying form. 



